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ABSTRACT 

A policy study was conducted in four rounds to assess 
the anticipated payoff of investment by the Advanced Research 
Projects Office in the development of a new family of terminals for 
use by the military in computer-assisted instruction. Members of the 
panel of experts proposed features^ rated them^ and reacted to the 
resulting ratings*. Of the 2U features, those rated as most needed 
iiere seen by th^se experts as lilcely to be in commercial production 
five to ten years from now in a form usable by the military. Experts 
generally agreed that investments in innovative pedagogical software 
and in innovative course writing are likely to have greater payoff 
than an investment in terminal developmant. Of 1U software features, 
those rated as most needed were ones for particularizing instruction 
online to the course-related needs of individual students. The report 
is a thorough description of the conduct of the study. (EHH) 
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20. ABSTRACT 

A policy study was conducted in four rounds to assess the anticipated payoff of 
an investment by ARPA in the development of new terminals for use by the military 
in computer-assisted and computer-managed instruction. Members of the panel of 
experts (II civilian, 9 military) proposed features, rated them, and reacted to 
the resulting ratings. Of the 24 features, those rated as most needed were seen by 
these experts as likely to be in commercial production five to ten years from now 
in a form usable by the military. Experts generally agreed that investments in 
innovative pedagogical software and in innovative coursewriting are likely to have 
a greater payoff than an investment in terminal development. Of 14 software 
features, those rated as most needed are ones for particularizing instruction online 
to the course-related needs of individual students. 

The report is a thorough^description of the conduct of the study. 
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Abstract 



A policy study was conducted in four rounds to assess the anticipated payoff of an 
Investment by ARPA in the development of a new family of termihals for use by the 
military in computer-assisted and computer-managed instruction. Members of the panel of 



axpertsTll civilian, 9 military) proposed features, rated them, and reacted to the resulting 
ratings* Of the 24 features, those rated as most needed were seen by these experts as 
likely tc be in commercial production five to ten years from now in a form usable by the 
military* Experts generally agreed that investments in innovative pjedagogical -software 
and in innovative coursewriting are likely to have a greater payoff than an investment in 
terminal development. Of 14 software features, those rated as most needed are ones for 
particularizing instruction online to the course-related needs of individual students. 

The report is a thorough description of the conduct of the study.* 



* A companion report by Louis Gallenson, An Approach to Providing a User Interface for 
Military Computer-Aided Instruction In 1980, ISI/RR-75-43; discusses effective utilization of 
forthcoming commercial terminals in military computer-aided instruction. 
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Executive Summary 



- A panel of experts (11 civilian and 9 military) was selected from the 
computer-assisted and computer-managed instruction (CAI/CMI) community. 

- Panel members were polled in order to determine what features they felt should 
go into innovative military CAI/CMI terminals and what payoffs they anticipated would 
foilow from these features. 

- The four rounds of questionnaire and feedback included: (1) general stance 
probing, (2) open-ended soliciting of features and payoffs, (3) ranking of features and 
.peyoffs,-and-(4) eliciting reactions to resulting rankings. 

- Participants rated investments [n either innovative pedagogical software oil 
innovative coursewriting as having higher potential payoffs than investments iji terminjj 
technology or large-scale use of existing systems. 

- The features felt to be most necessary in all CAI/CMI terminals are likely to be in 
commercial production by mainstream terminal vendors in a form usable by the military by 
1980-1985. 

- With respect to pedagogical software, the features felt to be most necessary are 
those that facilitate tailoring of interaction with a student to his particular course-related 
needs, interests, and difficulties. 

The report is a thorough description of the conduct of the study. 
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Introduction 



This report deals with one of many activities in a project attempting to provide 
functional specifications for a family of new CAI (Computer-Assisted Instruction) and CMI 
{Computer Managed Instruction) terminals for the 1980-1985, The purpose of the activity 
was to survey CAI/CMI experts in order to determine what features they felt should go 
into innovative military CAI/CMI terminals. 

The rationale for the survey was that innovative terminal design could be informed 
by sampling the projected needs and uses of the CAI/CMI user and research communities. 
It was assumed at the outset of the study that representatives of these communities could 
provide descriptions not only of desired terminal features, but also of anticipated payoffs 
from use of these features. 



The goals for the querying were to 

1. (1) Discover what features are desired in the military CAI/CMI terminal (or family 

of terminals) in 1980-1985. 

2. (2) Determine what the anticipated payoffs are for implementing and using these 

features. 

3. (3) Determine the importance of military CAI/CMI terminal development relative 

to developments in software, large-scale demonstrations, etc. 

4. (4) Determine the projected value of stand-alone versus networked systems. 



In order to carry out the survey, a panel of experts was selected by the study 
group working in conjunction with the ARPA Project Officer. The *panel members were 
surveyed using an iterative questionnaire technique described below. The procedures and 
materials used in the study are also described in a separate section below. Finally, the 
conclusions and recommendations resulting from the survey are reported. 
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Problems of Need Assessment 



Modern technology Is best characterized as immensely complicated, expensive to 
develop, and requiring long lead times from initial specification to mass production. Often 
the^cost of developing a new device is so high that potential users can afford it only if 
they are willing to guarantee a certain minimum order to the producer. Even without an 
agreement, the market ceases to act as a testing ground for determining user needs when 
demand is low relative to high initial development costs. Customers are forced to take 
what is available and like it. With long lead times and many innovative ideas that cannot 
be pursued profitably, it is not surprising that producers often fail to perceive the market 
correctly. It is in the best interests of all parties that representatives of the potential 
user communitic:* be involved in specifying the needs to which new technologies are 
supposed to respond. 

This report describes a methodology similar to Turoff's Policy Delphi which user 
communities might employ when establishing their needs.* It has been refined during a 
study of features of computer terminals necessary for CAI or CMI. 

Before attempting a ner#d assessment study, one should understand what is involved. 
The task is amorphous. Partly, it is an attempt to define the appropriate community. 
Partly, it is an attempt to discover the directions that the technology is taking. One hopes 
by shifting back and forth to discover the needs of a set of people which it appears to be 
feasible and extremely beneficial to satisfy. One can rarely go into the marketplace and 
survey needs, because consumers do not know what is or is not achievable by technology. 
In rapidly evolving technological fields, even those professionally involved have difficulty 
assessing what is achievable in a given time frame. An effort to determine needs is likely 
to be as much an educational process as it is an opinion-probing process. One of the 
greatest dangers is in assuming too much. User representatives are likely to possess 
widely differing competencies, interests, and skills. While there might be some advantage 
to having a randomly selected set of individuals rank order their needs, it seems mos,t 
reasonable that the user community representatives should be carefully selected for their 
knowledge and commitment. They should be allowed to interact over a period of time 
until they know enough about each other that they can work out compromises. Whllo It 
takes time, the process is likely to lead to recommendations that users can Mye. with and to 
which they can feel committed. 

The issues to be resolved are typically too complex and the number of factors to be 
represented is typically too great for a face-to-face confrontation to be productive. The 
Individuals who should be involved in the discussion are busy people who are rarely In the 
same place at the same time, and the priority-setting process takes more time than they 
can comfortably ^=^i^are. For the process to work, the user representatives must perceive 
that they have a great deal of influence over the final product without having to become 



» (M. Turoff, The Design of a Policy Delphi," Technological Forecasting and Social Cha n g e , 
1970,2, 149-171) 
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Involved in the mechanics of the process. When their advice is asked, they should feel 
that the questions asked are answerable and pertinent to the final recommendations. If 
they feel important points are being slighted, they should be able to alert the group. 
Unlike technology forecasting studies, it is more important to accurately represent and 
assimilate the opinions of group members than to strive for consensus. 

Turoff (1970) points out that at least three different participant roles exist in this 
type of policy polling — the individuals seeking policy advice, the small team that designs 
questions, compiles feedback, coordinates the effort, and writes the final report, and the 
larger group chosen to represent the various interests of the user community.* In our 
case, the Advanced Research Projects Agency's Human Resources Research Office 
requested a; study of what features should go into innovative military CAI/CMI terminals. 
The small grfoup that administered the study included a terminal designer, two experts on 
computer-assisted instruction, and three human communication researchers who acted as 
intermediaries between the large and small groqps. The large group included twenty 
experts or jfeams of experts educational researchers, CAI/CMI developers, instructional 
technologist's, computer scientists, and CAI/CMI courseware authors. 

In the following section, the procedures used for eliciting opinions during the various 
rounds are described. Slightly different approaches were used in each round. Questions 
were derived from responses and/or unresolved issues. We did not presume that 
statistical measures of significance could be employed in analyzing results. Instead, 
responses were condensed and organized in a way that merged similar opinions while 
preserving distinctive ones. It is unlikely that this group of participants would reach 
conclusions much different from the ones reported here if queried again. The reader will 
have to decide for himself whether the participants' preferences can be generalized to the 
entire CAI/CMI community. 



♦ Turoff, ofi. cit. 
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Procedures and Materials 

\ 

The basic procedure followed in this study was an iterative query and feedback 
technique using three questionnaires mailed to participants. Other materials provided to 
participants included working papers and scenarios, summaries of results from each round 
of questionnaire responses, lists and biographical sketches of participants, and 
miscellaneous administrative correspondence. These materials are contained in the 
appendixes. The materials and specific procedures are described below. 

Selection of Participants and Initial Mailing 

Potential participants were selected by the authors working in conjunction with Dr. 
William Mann, the principal investigator of the overall project at the Information Sciences 
Institute, and ARPA project officers, including Col. Austin Kibler and Dr. Thomas 
O^Sullivan. Nine participants or participant teams were selected from the military 
community. Twenty-five potential participants were selected from the civilian community 
(business and education) in the expectation that about half would participate. It was 
hoped that there would be an approximate balance of participants from the military and 
civilian communities in the final participant pool. This balance was achieved (see Table 1). 
Most of the potential participants wer.e contacted by telephone by one of the authors. A 
letter confirming telephone responses and/or soliciting participation in the study was sent 
to potential participants on August 30, 1974 (Appendix II.A). Enclosed with this letter 
were a working paper describing the study and a questionnaire requesting biographical 
information (Appendix II.A). A list of potential participants being contacted was also sent 
in this mailing. Different cover letters and participant lists were sent to potential civilian 
or military participants. Responses trickled in, and a few participants nominated other 
individuals either in substitution for or in addition to themsjilves. Letters were sent the 
week of September 31, 1974, acknowledging receipt of bjographical questionnaires (which 
confirmed the participant's intention to participate in \he study) or rf»questing completed 
questionnaires from a few participants who responded by lette*- nnly . 

A list of the actual participants with notations regarding questionnaire rounds in 
which they participated is contained in Table 1. A tottl^gt^S experts agreed to 
participate. Two never responded further. The balance is ^distributed into 20 
participant-teams, as noted in Table 1. Brief biographic^ summaries Of active participants 
are contained in Appendix I.A. In addition, a note concerning Solicitation of military 
participants is contained in Appendix I.B. 
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PARTICIPANT 



Dr. Ernest Anastasto 
Dr. Alfred BorK 
Dr. Pater Dean 
.Mr. Richard Ditzik 
Dr. Robert Fitzhugh and 

Dr. Robert Glaser 
Dr. John Ford 
Mr. Frank Giunti 
Dr. Keith Hall 
Dr. Albert Hickey 
CPT Larry Hinkle 
Mr. Donald Kimberlin 
Dr. David Merrill 
Dr. Leon Nawrocki 
Dr. Marty Rockway 
Dr. Worth Scanland 
Dr. Bruce Sherwood 
Dr. Joseph Ward 
MAJ D. A. Weihe, 
CPT D. Glessner, and 

TS6T L Miller 
Dr. Jon Wexler 
Dr. Karl Zinn 



STATUS KEY: 
^Military 

oParticipated as a "team**. 



TABLE 1 

Active Participant Response Record 

ROUNDS 

STATUS ONE TWO THREE FOUR»» 
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9 



♦♦All participants were telephoned in Round Four. 
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Questionnaire Rounds . 

Each round of questionnaires was designed by the authors. The questionnaire for 
uach round of query after the first round was based upon the responses to the previous 
round Participants were sent an introduction to each round and a summary of responses 
to the previous round for each round after the first. 

Questionnaires and summaries are contained in Appendix III. 

Round One. The questionnaire for round one was designed to probe the general 
attitudes or "stance" of each participant regarding CAI/CMI, to discover projected military 
uses of CAI/CMI, and to direct query and feedback during future rounds. Each question 
provided two or more alternatives and asked for the respondent's preferred 5»tance as 
well as stances he would be willing to accept for purposes of consensus. While consensus 
(i.e., a preponderance of preferences for one alternative) was achieved on some points, 
the exercise was primarily useful in identifying and clarifying potential issues which could 
be raised in Round two. 

Along with the questionnaire, a set of scenarios was provided to orient participants 
by suggesting future CAI/CMI environments in the military (Appendix II.B). The authors 
expected that participants with no military experience would use the scenarios as a 
surrogate for such experience, and that participants with military experience would 
suggest revisions of the scenarios. Since no comments about the scenarios were received 
from the participants, the scenarios were not referred to again in the course of the study. 
As shown in Table 1, 17 responses were received in round one. 

Round Two. The questionnaire for round two was developed after receiving the 
majority of responses (16) from round one. Round two was used as a bridge between the 
first round (probing participants' stances) and the third round (concentrating on specific 
terminal features). Participants were asked to respond to three types of questions: 

(1) Anticipated payoff of investment in terminal development; 

(2) Identification of terminal features to be explored in round three; and 

(3) Continued probing of round one Issues, such as design of terminals for users 

with a wide range of motivation and intelligence levels. ^ 
The questions were open-ended and succeeded in eliciting substantial writtert^ponses. 
It was possible to sense informally which issues were felt to be significant and to feed this 
back to participants in both the summary and in the round three questions. 

In round two participants were encouraged to respond by an offer of a small 
stipend. Civilian participants who responded did receive the stipend. Unfortunately, it 
was later realized that participants employed by the military could not be paid from 
project funds; so, a letter informing them of this problem was sent to those participants 
(Appendix 11.0). Nevertheless, 7 of the 9 military participants responded to round two, and 
9 of the n civilian participants responded 
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Round Three. Following the previously established procedures, the round three 
questionnaire was designed and sent to participants along with a summary of round two 
results and a list of active participants. The questions were structured so as to get 
degree of commitment information regarding particular policies snd features. The lack of 
qualifying remarks in the responses suggests that the questions were appropriate and the 
. respondents felt they were ready to vote. 

The analysis of the third round questionnaire responses constitutes the results 
section of this report. A total of 14 responses were received in round three. 

Round Four. In a final mailing this report in draft form was sent to participants with 
the request that they communicate any comments and/or dissenting opinions to the 
authors. Participants were also contacted by telephone. Dr. Merrill said that he thought 
the emphasis on course development rather than terminal development was consistent with 
his feelings. Dr. Wexler stated that he expected that the final round would have resulted 
in a more precise specification of desirable terminal features, with relative weights. Mr. 
Ditzik said he felt the study should be replicated with a wider selection of participants. 
Six other participants responded with no further comments. 
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Results 



The major conclusion of the study is that research money should be invested in 
pedagogical software or coursewriting rather than m the development of new military 
terminals. It is shown that the terminal needs of both civilians and military should be 
combined, that investing in new terminal development receives low priority, that features 
of terminals felt necessary will most likely be available from commercial vendors, that no 
resolution of the stand-alone/time-sharing problem can be reached, and that software 
features felt most necessary are interactive ones for tailoring instruction to the 
course-related needs of individual students. 



1. It is in the best interests of advancing the CAI/CMI terminal state-of-the-art to 
combine civilian and military needs as opposed to focusing on just military or iyst civilian 
needs. When asked this question, ten respondents favored combining military and civilian 
needs, zero favored keeping the needs distinct, two chose the category *'other", and two 
did not respond. The two "other" responses both indicated that there are some unique 
military requirements and instructional situations requiring special terminals, but that 
generally military and civilian needs can be combined. Almost every participant at some 
point during the study observed that special terminals are needed in special situations. 
Sometimes ruggedness and/or portability are essential. Other times simple input not 
involving typing and/or verbalizing is important. 

2. !n terms of their potential payoffs for advancing CAI/CMI state of the. art 
(assuming a 1980-1S85 time frame), investments in either innovative pedagogical software 

or: innovative coursewriting rank higher than investments in terminal technology. When 
asked to rank four possible investment strategies from 1 (greatest potential for advancing 
the state-of-the-art) to 4 (least potential), terminal hardware received an average rating 
of 3. Both an investment in software and an investment in coursewriting received an 
average rating of about 2. The fourth alternative — an investment in large-scale use of 
existing systems — averaged out about equal to terminal hardware. Exact tabulations are 
included in Appendix III.F. 

It should not be assumed from this ranking that an investment in terminal hardware 
Is not considered important. In the second round at least eight participants felt that an 
investment in terminal development would have a significant payoff. Many of the reasons 
given in support of such an investment argued that the CAI/CMI marketplace was not large 
enough to attract special attention from vendors and that business-oriented terminals 
would not meet instructional needs. T^is concern for the size of the marketplace may 
explain why it is in the best interests of civilians and military to combine their markets. 



3. Itie features felt to be most necessary ijh aN CAI/CMI terminals are Ijkely to be 
available from mainstream terminal vendors. When members were asked to rate 24 
possible terminal features in terms of their necessity using the following rating scheme 
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++ definitely yes 
would be nice 
0 neutral 
- probably not 
— definitely not 

seven features received an average rating between ++ and +. [See Appendix III.F, 
question 4(1) for details.] The two most definitely needed features (alphanumeric typed 
Input and k single action to invoke frequently used functions) are available on most 
existing te/minals. Only limited graphics features are definitely needed (an ability to touch 
Or poinLTO screen locations and an ability to generate simple straight line figures). 
Symbpt set requirements may not currently be available in many terminals, but will not be 
too difficult to incorporate (programmable symbol sets and a variety of predefined symbol 
sets). The final definite need is for signals for controlling external equipment, which is 
dl$o not too difficult to incorporate. 

On the other hand, the only features with "njjutral" or below ratings are precisely 
those that might prove difficult to implement (spoken word input, shading and texture in 
display, and generation of large numbers of symbols in displays). 

The remaining features fall generally in the neutral to "it would be nice" range. At 
the lower end of the range are c6lor displays, memory, exact reproductions of screen 
Images, computer-composed speech, and lights under keys. At the upper range are 
signals received from plug-ins, alphanumeric printouts, line drawing input, display of 
complex line drawings, high- and low-resolution stored visuals, moving visuals, 
prerecorded audio output, and processing capability. 

It is the considered opinion of the investigators that by the time a terminal could 
move from development to large scale production, commercial vendors will be marketing 
terminals that meet all but perhaps the least necessary of these features. 

4. No recommendation can be made about whether terminals should be stand-alone 
CAI/CMI systems or rather communication devices connected to time -sharing systems, jin 
both the first rouna and in the third round, participa^fits were asked about whether or not 
terminals should be stand-alone systems. In the first round 9 ranked stand-alone over 
time-shared with 5 ranking the options the other way. In the third round 3 felt 
stand-alone capability was definitely needed, 6 felt it would be nice, 4 were neutral, and 1 
was slightly opposed. While both responses show^ a preference for stand-alone systems, 
the trend is not strong enough to be called consensus. A glance at the comments in 
Appendix III.D will reveal why some participants feel very strongly in favor of stand-alone 
systems. Perhaps the best solution is to rely upon the commercial marketplace, which is 
likely to provide a variety of solutions to the stand-alone problem. 



5. Turning to pedagogica l software, the features felt most necessary [n CAI/CMI 
software are course-related « st udent -tailored and interactively oriented. When asked to 
rate fourteen possible software features in terms of their necessity using the following 
rating scheme. 
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++ definitely yes 
•f would be nice 
0 neutral 

- probably not 

— definitely not 

four features received an average rating between ++ and +. [See Appendix IILF, question 
7(1) for details.] The most definitely needed was software permitting instructional 
sequences tailored to the abilities and/or weaknesses of individual students. Next came 
problems and examples responsive to interests of particular students, hints that reduce 
problem difficulty, and summaries of student progress. 

Immediately following these features with average ratings of + or close to + are 
course-related '•intelligent'' features that (1) contrast concept/strategy information with 
information about potential students, detect difficulties, and advise the instructor; (2) 
discover patterns of course-related behavior and advise the instructor; (3) derive 
course-related strategies from examples provided by the instructor; (4) respond 
meaningfully to course-related problem-solving strategies; (5) respond meaningfully to 
course-related questions or statements by students; and (6) accumulate course-related 
concepts and vocabulary from examples provided by the instructor. 

Finally with ratings between + and neutral are features that respond to (1) 
course-independent problem-solving strategies; (2) pauses; (3) dialogue cues; and (4) 
course-independent questions. 

While it was not the major intention of the study to probe into specific software or 
courseware areas needing attention, it was felt from the results of round two that 
exploratory probing might prove fruitful. Unlike the list of terminal featuf^, which was 
assembled from round two recommendations, the software list was assembled purely by 
the investigators. No claim is made that the best or most relevant features were included. 
Most likely a better analysis could be done at some future time. However, when asked for 
additional software features, only five of the fourteen respondents mentioned anything. 
Two mentioned that they had worked so long on the problem that it was hard to 
summarize their opinions. Two others mentioned the problem of exporting courseware for 
use on other systems. For more comments regarding authoring and adaptable software, 
see the responses to question 3 of round two in Appendix IILD. 

In conclusion, it would seem that the major focus for funding should center upon 
software and/or courseware and not upon terminal hardware. Perhaps people in the 
computer-based instruction field are recognizing that hardware cannot solve the problem 
of good instruction, Rockway summarizes: 

•The basic problem of so called teaching software is that most of the 
^material* encountered ^akes no attempt to understand the student^s 
Knowledge except as reflected by simple answers to multiple choice questions. 
The material does not carry out the dialogue of an expert tutor because most 
authors do not understand how this is done or because it cannot be 
supported by the computer available to the author, „, We could probably 
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profit both from development of the technology and technique of carrying out 
verbal and manual dialogue. The case for an expensive electronics 
presentation system for fixed material is very difficult to make,** 
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Appendix LA ) 
BIOGRAPHICAL SUMMARIES OF PARTICIPAI^fTS 



DR. ERNEST J. ANASTASIO 

Associate Director 

Data Arr^Hysis Research Division 

Educatmnal Testing Service 

Princeton, NJ 08540 

(609)921-9000 

Director of Educational Technology Research and Associate Director of 
the Office of Data Analysis Research at ETS. 

He has taught graduate studies in statistics and the use of computers in 
research at'Princeton University and the New School for Social Research. His 
research interests are in areas of instructional technology, computing 
methodology and the methodology of modern data analysis. 



DR. ALFRED M.BORK 
Department of Physics 
University of California 
Irvine, CA 92664 
(714)833-6665 

We have developed at Irvine, a large group of student-computer dialogs 
for physics use, including the underlying macro-based software. Most of the 
material uses graphics, with the Tektronix 4013 or our primary terminal. 
These materials are in heavy use in our beginning classes, and are beginning 
to see some use on other campuses of the University of California. 



DR. PETER M. DEAN 
IBM Corporation 
3424 Wilshire Blvd. 
Los Angeles, CA 90010 
(213)382-7272, ext.1272 

Ed.D. Columbia University - Teachers College Science Education. 
Manager Technical Requirements, EDEX teaching - systems. Manager 
Interactive Terminal Education Development, IBM corporation. 



MR RICHARD DITZIK 
Control Data Corp. 
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BLNtSD 

8120 Pann. Ave., S. 
Bloomington, Minn. 55431 
(612)633-0371, ext.391 

BSE (EE), University of Michigan; M.S. Cybernetic Systems, California 
State University, San Jose. Presently representing Control Data Corporation 
(Terminal System Division) efforts in developing an education terminal for 
future .CBE systems. Primary interest in computer-base instructional 
commurflcation systems.^ 

DR- ROBERT FITZHUGH 
LROC 

University of Pittsburgh 
Pittsburgh, PA 15213 
(412)624-4895 

Computer scientist interested in system design and educational use of 
computers. 

DR. JOHN a FORD, JR. 

Advanced Instructional Systems Directorate 

Naval Personnel Research and Development Center 

San- Diego, CA 92152 

(714)225-7121 or 7140 

Assoc. Director, Advanced Instructional Systems, Navy Personnel 
Research and Development Center, San Diego. Ten years experience at Navy 
Personnel and Training Research Lab., San DiegO; SDC, 1958-64; RAND 
1956-58. Academic experience: Temple University and University of 
Delaware. Education: Ed.D. Teachers College, Columbia University 1954. 
Research interests: Instructional research and technology development 
including CAI/CMI and simulation. 



MR. FRANK E. GIUNTI 

Commander, U.S. Army Training Support Activity 

Attn: ATTNG-PA-TS 

Mr. Frank Giunti 

Ft. Eustis, VA 23604 

(804)878-5801 

Mr. Frank E. Giunti has been serving as the Technical Director, 
Computerized Training Systems Project, Product Manager's Office, since its 
establishment in August 1972. Prior. to this period of time he served as the 
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Chief, CAI Division, US Army Signal Center and School (October 1970 to 
August 1972) and initially as an instructional programmer, CAI classroom 
supervisor, and CAI project planner (August 1966 to October 1970). 



CAPT D. GLESSNER 
AFMPC/DPMYC 

Randolph Air Force Base, TX 78148 
(512)652-2414 

BS in business administration. State University of New Yorkj MBA, 
University of Alabama; 8 years active duty with the Air Force. Teach 
management at San Antonio College. AFMPC representative on study of 
Automatic Processing Requirements of the 80's (SADPR-85) and the Base 
Communications Mission Analysis (BCMA); developed and implemented 
research method to analyze base level functional requirements. 



DR. ROBERT GLASER 
LRDC 

University of Pittsburgh 
Pittsburgh, PA 15213 
(412)624-4895 

Psychologist interested in instructional research'^dM development. 



DR. KEITH A. HALL 

College of Education 

The Pennsylvania State University 

201 Chambers Building 

University Park, PA 16802 

(814)865-0471 

Graduate study in instructional systems and technology and educational 
psychology - Indiana University. Research interestes in adaptive, interactive 
instructional systems. Management responsibilities for 4 CAI systems - 1 
fixed site and 3 mobile systems. 



DR. ALBERT E. HICKEY 
Entelek Incorporated 
42 Pleasant St. 
Newburyporl,,MA 09150 
(617)465-3000 
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President of Enteiek, Inc. Consultant to ETS and ARPA. Author of 
Research Guidelines for CAI. Background in experimental psychology and 
human engineering. Author of PI & CAI programs, especially for industry. 



OPT LARRY HINKLE 

Commander, US. Army Training Support Activity 

Attn: ATTNG-PA-TS 

OPT L Hinkle 

Ft Eustis, VA. 23604 

(804)878-5801 

No biographical information received. 



MR. DONALD A. KIMBERLIN 

GTS Field Office 

TRADOC 

Project ABACUS 

Signal Towers, Room 709 

Ft. Gordon, GA 30905 

(404)791-3193 or 7297 

Mr. Donald A. Kimberlin served as an instructional programmer, 
course development team chief, and classroom supervisor in the CAI Division 
from 1968 to 1972. From 19>2 to the present, Mr. Kimberlin has served as 
an Educational Specialist and Chief of Course Development for the CTS 
Project. He is now the Chief of the Course Development and Applications 
Division, Project ABACUS. 



DR. DAVID MERRILL 
Department of Education 
Brigham Young University 
Provo, UT 84601 
(801)224-2350 

My Ph.D. was obtained from the University of Illinois under Larry 
Stolurow, my dissertation being one of the first studies on SOCRATES. I have 
published in the area of instructional design. I was leader of the team which 
did courseware design for the TICCIT system. This design was based largely 
on the theoretical work which I had done on instructional design. I am 
currently on sabbatical leave from B.Y.U. and serving as Vice-President of 
Courseware, Inc. which is currently involved in two major projects training 
military personnel to develop CAI materials. 
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TSGT L A, MILLER 
AFMPC/DPMYC 

Randolph Air Force Base, TX 78148 
(512)652-2414 

Entered the Air Force in 1955; participated in the tests the Air Force 
was conducting prior to release of CAI Air For.© wide in 1972; has directed 
training programs and managed CAI/CMI systems. 



DR. LEON H. NAWROCKI 

Computer Instruction Research Program 

US. Army Research Institute 

Commonwealth Building 

1300 Wilson Blvd. 

Arlington, VA 22209 

(202)694-3954 

Dr. Nawrocki received his Ph.D. from the Ohio State University in 
1969 and has since been employed by the Army Research Institute. From 
1969 to 1972 he was assigned to Command Systems unit and conducted 
research on information displays. From 1972 to present he has been senior 
psychologist in the Educational Technology unit. Organizational membership 
includes APA (Divs. 1 & 21), HFS, ADC\S and AERA. 

DR. MARTY R. ROCKWAY 

Technical Training division 

Air Force Human Resources Laboratory 

Lowry Air Force Base, CO 80230 

(303)394-4385 

Marty R. Rockway is a native of Chicago, Illinois. After completion of 
undergraduate work in the physical and engineering sciences he received a 
Ph.D. in psychology and statistics from Northwestern University in 1953. In 
1963-64 he was a Princeton Fellow at the Woodrow Wilson School of 
Princeton University where he majored in the areas of public administration 
and national security affairs. During 1967-69 he was a Littauer Fellow 
engaged in a joint program in management science and science and public 
policy at Harvard and M.I.T. 

During the past twenty years Dr. Rockway has held a number of R and 
b posts within the Air Force Systems Command, including the position of Chief 
Engineer for Human Factors at the Aeronautical Systems Division and his 
current position as Technical Director, Technical Training Division, Air Force 
Human Resources Laboratory. 
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DR. WORTH SCANLAND 

Chief of Naval Education and Training 

Code N-330 

Naval Air Station Pensacola, Fla. 32508 
{904)452-3466 

Attended the Naval Academy followed by 30 years active duty as a 
Naval officer, mostly submarines, followed by graduate, studies at FSU, with a 
M.S. in educational research and a Ph.D. in Instructional Technology, 
followed by a couple of years as director of research with the Florida Youth 
Services Authority (juvenile delinquency), followed by current duties with the 
Naval Education and Training Command staff. 



DR. BRUCE SHERWOOD 
Room 252 

Engineering Research Lab 
University of Illinois 
Urbana-Champaign^ IL 61801 
(217)333-6210 

B.S. Engineering Science, Purdue Univ.-1960 

Fulbright to Padova, Italy 
Ph.D. Physics, University of Chicago - 1967 

Taught and did experimental particle physics research at Caltech 
1966-1969. At University of Illinois (Urbana) since 1969 - now Assistant 
Director of the Computer based Education Research Laboratory (PLATO) and 
Associate Professor of Physics. Worked on design and implementation of 
TUTOR language. Developed PLATO version of introductory classical 
mechanics course. Author of the text The TUTOR Language. 



DR. JOSEPH S. WARD 

U. S. Army Research Institute 

Commonwealth Building - Room 2045 

1300 Wilson Blvd. 

Arlington, VA 22209 

(202)694-1397 



Dr. Ward has worked in the design, development, management, and 
evaluation of CAI/CMI systems in Army training programs. His primary 
interests in this study are in instructional systems development research 
involving CAI/CMI as one delivery mode of instruction. 
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MAJ D. A. WEIHE 
AFMPC/DPMYC 

Randolph Air Force Base, TX 78148 
(512)652-2414 

* I have a degree in secondary education from the University of Wichita 

(now Wichita State University) and 16 years experience in Air Force 
personnel management. I also have 27 hours of graduate study leading to an 
MS in systems management from St. Mary's University, San Antonio, Texas. 



DR. JON WEXLER 
(During study) 

Department of Computer Science 
State University of New York at Buffalo 
4226 Ridge Lea Rd. 
Amherst, NY 14226 ' 
(Currently at Tempe, AZ) 
(602)967-3248 

Research/teaching interests are in the area of artificial intelligence and 
the application of its representations and processes to computer-based 
teaching systems to generate portions of the material needed for intelligent 
(interesting) student-computer dialogue. Involved in the (slow) development 
of a generative teaching system for multiple programming languages; current 
work focuses on the generation of equivalent target language programs from 
a visually-oriented ab3tra:t language. 



DR. KARL L. ZINN 
Research Scientist 
University of Michigan 

Center for Research on Learning and Teaching 
109 E. Madison Street 
Ann Arbor, Ml 48104 
(313)763-4410 or 0158 

Karl L Zinn, Research Scientist at the Center for Research on Learning 
and Teaching, and Associate Director of the MERIT Computer Network at the 
University of Michigan, is engaged in development of innovative uses of 
computers in education, giving special attention to computer languages and 
supportive systems. He has worked with dozens of curriculum authors in a 
variety of subject areas who have prepared learning exercises using ten 
different authoring languages as well as a number of general-purpose 
programming languages. 
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Appendix LB 
MILITARY PARTICIPANT SELECTION 

MEMORANDUM FOR THE RECORD 

From: Monty C. Stanford 
Date: October 28, 1974 

Subject: Obtaining Military Participants in the CAI Study 



Military participants were obtained for the CAI terminal study in the same manner as 
civilian participants. That is, the same kind of letter requesting participation in the study 
was sent to military participants as was sent to civilian participants. Names of military 
participants were obtained from recommendations and by contacting key agencies in the 
various armed services. 

One reason for using this method was to attempt to ensure that a participant from one of 
the armed services was not participating merely because he or she had been assigned by 
his or her superior to participate. I still feel that such "volunteer** participation is good 
and is a desirable aspect of this and future studies. 

However, several military personnel e.xpressed some hesitancy to participate in the study. 
They seemed to be unsure as to whether or not such participation would be sanctioned, 
approved, or required by higher command. And, in one case, the potential participant we 
contacted desired to participate in the study, but, after receiving our initial materials, the 
person*s superiors denied permission to participate in the study. 

* 7 

Recommendation: 

It is, therefore, recommended that in future studies of this type participation be requested 
from military personnel on a volunteer basis in the same manner as was done in this study. 
But, it is also recommended that the highest command level possible be contacted and 
approval for participation in the study be obtained before requests are made to individual 
military personneL 
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Appendix IIAl.a 
COVER LETTER TO CIVILIAN PARTICIPANTS 



August 29, 1974 



[Name of participant] [Address of participant] ' 

Dear [Title and last name]: 

The Information Sciences Institute and the Annenberg School of Communications at 
the University of Southern California are conducting a study for the Advanced Research 
Projects Agency in order to learn from experts what features should be included in 
pace-setting CAI terminals five to ten years from now and why. The intention of the 
study is to discover if there are new devices and/or strategies for making CAI user 
interfaces more effective. 

Would you be willing to participate on a panel of experts from October 1974 until 
January 1975? We expect to conduct four rounds of query and feedback regarding 
possible features and reasons for the features. Enclosed is a working paper detailing the 
goals of the study and procedures we plan to follow, a questionnaire that will help us all 
better understand each other, and a list of the other people who are being contacted 
Please let us know as soon as possible whether or not you can participate and who else 
we ought to contact. 



Enclosures 



ERIC 



Sincerely yours, 

/s/Monty Stanford 

for/ Bill Mann, ISI 

Rick Carlson, ASC 
Tom Martin, ASC 
Monty Stanford, ASC 
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Apperiai>iriLA.Lb 
COVER LETTER TO MILITARY PARTICIPANTS 



^Name of participant] 
[Address of participant] 

Dear [Title and last name]: 

in reference to our recent telephone conversation, I am enclosing materials on the 
Computer Assisted Instruction Terminal Study being conducted by the Information Sciences 
Institute <ind the Annenberg School of Communications at the University of Southern 
California for the Advanced Research Projects Agency. We hope to learn from expert^ 
what features should be included in pace-setting CAI terminals five to ten years from now 
dnd why. The intention of the study is to discover if ther^ are new devices and/or 
strategies for making CAI user interfaces more effective. 

Through the course of the study^ we expect to condcJct four rounds of query and 
feedback regarding possibley features and reasons for the/features. Enclosed is a working 
paper detailing the goals of the study and procedures we plan to follow, a questionnaire 
that will help us all better understand each other, and a list of the other people in the 
military community who are being contacted. We have also enclosed brief vitas of the 
ASC study members, so that you will have some idea of who we are* 

In addition to participants from the military community, we are corttacting potential 
participants from academic and business organizations who are CAI users or researchers. 
Brief biographical statements on all participants will be sent to you with the first round of 
questionnaire materials. 

We look forward to your participation in this study. Please let us hear from you 
as soon as possible. 



Sincerely yours, 

/s/Monty Stanford 

for/ Bill Mann, ISI 

Rick Carlson, ASC 
Tom Martin, ASC 
Monty Stanford, ASC 

Enclosures 



Policy Study on CAI Terminals 
Appendix l!A2 - Working Paper 



27 



Appendix II.A.2 
WORKING PAPER 

The ARPA Contract 

There are a number of tasks included in the contract between the Information 
Sciences Institute and ARPA*s Human Resources Research Office, only one of which is the 
querying of experts. Some of the other projects that will be completed during the first 
year include 1) putting PLATO terminals on the ARPA network, 2} surveying the literature 
for CA! user interface descriptions, 3} tracking technological developments that might 
significantly advance CAI terminal state-of-the-art, 4) transforming the recommendation of 
the panel of experts into specifications for a terminal that can be sent out to contractors. 
The major long range study is directed toward discovering and modeling human discourse 
processes that can later be incorporated into interactive computer systems. 

Goals for the Querying of Experts Task 

Available terminals frequently limit what authors, designers, and researchers can do 
with CAI or CMI systems. In order to have equipment on hand five years from now that 
meets needs then, we must find out what designers, teachers, and researchers plan to be 
doing and what they will need in a terminal. While we must produce an end product that 
can be turned into specifications, we wil! not limit ourselves to terminals. If it is more 
important that certain types of software be developed, or that new types of learning 
laboratories be established, we want to find that out. We are particularly interested in 
what terminat feaiures that are not normaiiy needed are needed for carrying out user 
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Interaction research. We suspect many of you have good ideas about data capturing 
techniques, monitoring devices^ and plug-in features you feel you need but which cannot 
b0 justified in mass-produced terminals. However, if discussion becomes too blue sky, we 
plan to move back toward operational environments. We are expected to make 
recommendations regarding stand-alone versus networked systems, but will fight hard to 
keep this from becoming the sole topic of discussion. 

It IS very important that we not como up with a list of features without filling in the 
reasoning behind the features. We want to know what payoffs you see in the features 
you recommend While it may not be possible to justify features in a strict cost/benefit 
sense, curiosity alone Is probably not enough. We hope you use the justification process 
as an opportunity for influencing each other. 

One final note: ARPA is interested in helping to make military education more 
efficient and effective. While the recommendations we come up with are likely to have 
wide applicability, they must take into consideration the type of student, teacher, and 
learning environment encountered in the military. We plan to provide background material 
to those of you who are unfamiliar with Armed Services education. 

-Procedures We Plan to Follow 

In many respects we are planning to have the study be like a Delphi study. There 
will be a series of rounds, feedback will be used to bring participants towards consensus, 
names will not be mentioned, and most of the interaction will be through the mail. 
However, we are not interested in finding out when you think some development will take 
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place, but rather what you want to have done. In the early stages we plan to be frankly 
nonstatisticaL We are more interested in letting you inform each other about your 
positions than in forcing you to react to ours* In order to help you we plan to provide 
you with background scenarios, sample position statements, and checklists of things about 
which others might want to know your opinions. In the first few rounds we are really 
looking for statements that will bolster the final selection of features. 

The final rounds are intended to be much more concrete and feature*oriented You 
will be asked to rate how important you feel various features to be, and to agree with or 
attempt to reword supporting reasons. We expect that some of you will feel the need to 
talk things over with us or with each other. We plan to be available via the telephone, 
can make a limited number of site visits (provided there are clear and compelling reasons), 
hope to use teleconferencing via the ARPA network and/or the PLATO network, and may 
need to hold a one-day workshop in order to reach consensus. We welcome suggestions 
from you regarding how to improve upon the querying procedure. 

The final procedural issue is one for which we have no good answer— how to come 
up with a single set of recommendations. A number of participants are bound to feel 
there i$ a need for a number of different types of terminals. Other participants will feel 
that features cannot be talked about in isolation. We may be forced to come up with a 
family of terminals each responding to a different environment^ or a family ranging from 
cheap to expensive. Features are likely to cluster into groups, v/ith some groups less 
essential than others, and some groups mutually exclusive. The match between features 
and reasons for features is not likely to be one-to-one. However, we will do our best to 
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keep participants working toward a loosely ranked list of justtfted features for a single 
terminal. 

If you have any further questions, call Rick Carlson, Tom Martin or Monty Stanford 
ft (213) 745-6273. We want you to feel that the study is for your benefit more than for 
ours, and hope that you will let us know how it can be made more responsive to your 
needs. We think the procedure will work and that the potential payoffs are great enough 
to make it a worthwhile endeavor for everyone involved. 
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Appendix ILA,3 



BIOGRAPHICAL QUESTIONNAIRE 

In order, to give all of us who are participating in this study some idea of the 
composition of the particip::nt group, we would appreciate your completing and returning 
to us this biographical questionnaire. 

If you have, in addition, a one-page vita or references to papers or articles that you 
feel reflect your current thinking on the topic of this study, it might be helpful for you to 
include these also. 

1. Your experience relative to computer assisted or computer 
managed instruction: 

1.1 In general it would be helpful to Know what experience 
you have had working with CAI/CMI systems that you 
feel is relevant to the topic of this study. But 
first, there are some specific items of information 
that we would like to obtain from all participants. 
For these items, would you please check the appro- 
priate items beiow. For each item> circle CAI. CMI, 
or both. 

( ) I have taken courses using CAI/CMI systems. 

< ) I have authored CAI/CMI coursework. 

( ) I have designed and/or programmed CAI/CMI programs. 

( ) I have managed a CAI/CMI system. 

( ) I have designed terminal hardware. 

( ) I have other relevant experience, including: 



1.2 Now, could you describe your working experience that 
you feel is relevant to the topic of this study? 



c 
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2. Your experience relevant to military training: 

2.1 Again, since the ultimate goat of this study is to 

provide recommendations for CAI systems in the military 
training environment, it would be helpful to Know what 
kind of familiarity you have with that environment. 



( ) I have been a studenKin a military service 

training cour$e. 
( ) 1 have taught in a military service training program. 
( ) I have authored military training materials. 
( ) I have directed a training or education program 

at the local command level or higher. 
( ) I have worked on the staff of a military service 

training command (or at DOD level). 
{ ) t have other experience with military training, 

including: 



2.2 Now, could you describe your working experience with 
military training that you feel is relevant to the 
topic of this studyt 



3. Your avalUbifity and access to computer networks: 

3.1 Are there any times during the conduct of this study 
when you will not be available? 
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3.2 If you have access to the PLATO network or to the 
Advanced ResearchtProjects Agency (ARPA) network, 
it might be most convenient to communicate with you 
via one of the networks. D6 you have access to 
either or both of these networks? 

( ) PLATO 
( ) ARPA 



3.3 If you havev access to either or both of these networks, 
would prefer to participate in the study via one 
or the other? 



4. Your suggestions and comments on this study: 

41 What benefits do you anticipate or would you like to 
derive from participating in this study? 
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Do you have any comments on the design and procedures 
for this study and/or do you have any suggestions for 



4.2 

the study? 



5. In the first round of this study we will provide all 
participants with brief (say, five lines) biographical 
summaries of all participants. What would you like to 
say in yours? 
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Appendix il.B.l 
INTRODUCTION TO ROUND ONE 



As you know from reading the working paper (distributed at the beginning of the 
study), the end product of the querying process must be translatable into specifications 
for future CAI/CMI terminals. In this round we are hoping to find out what some of your 
general attitudes are, how you see CAI/CMI fitting into the military environment, and in 
what areas we can direct discussion during future rounds. Enclosed you will find a set of 
nine scenarios and a questionnaire. The scenarios are intended to orient participants by 
suggesting (not delimiting) how CAI/CMI terminals might be used in 1980-1985. They 
hopefully will act as a surrogate for military CAI/CMI terminal projected usage data. The 
questionnaire contains eight general attitude questions and fifteen usage environment 
questions. We do not view the questionnaire as a validated instrument for gathering 
reliable data. Rather we view it as a sounding board for stimulating and focusing 
discussion. Notice the wide empty marair: next to questions. We want to know what the 
questions mean to you and how you would follow up on them in succeeding rounds. If you 
have no attitudes about an issue, do not (§el compelled to plac e an V next to one of the 
alternatives. If your attjtude has stipulations , tell us what those stipulations are. 

Some participants have asked why we are diverging from the Delphi methodology 
and whether or not we view our methodology as reliable. From a social scientific view, 
the querying process we are employing cannot be called reliable — the sample size is too 
small, we are querying experts rather than end users, the participants were not chosen at 
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random from an homogeneous population, and the questions have not been thoroughly 
pre-tested. Nevertheless, we believe that it is belter to ask your opinions than to 
pretend we already know all we need to. Quexwig panels of experts is a common 
practice when establishing standards for hard and soHv technologies. If we all keep in 
mind the limitations of the methodology and the vastl^ differing backgrounds we come 
from, perhaps we can educate each other. 

Delphi studies have come under criticism recently in a report by Harold Sackman of 
the Rind Corporation ( Delphi Assessment; Expert Opinion, Forecasting, and Group Process, 
K Sackman, The Rand Corporation R-1283-PR, April 1974, 117 pp.). We are attempting 
to obviate some of his objections to Delphi. He objects to the estimating of future dat.es, 
which we are not attempting to do. He argues that the anonymity of participants leads to 
a lack of responsibility — we plan to s^ummarize where possible, but not to guarantee 
anonymity. He objects to lack of supportive reasoning behind predictions, so we plan to 
farret out the reasoning of each participant regarding the more significant questions. 
While we are hoping for consensus, we do not plan to punish outliers, and instead plan to 
bring their arguments to the attention of all so that the most persuasive reasoning can 
prevaiL 

Feel free to challenge us on the methodology. If you think the profiles are a waste 
of time, tell us. If yOu think thsy could be revised to really get at the essence of military 
Instruction, revise them or make up your own. If you think questions are unclear, 
ambiguous, or too "lumpy", please suggest revisions. The second round will cover terminal 
input/output features, but the third round will return to the topics covered in round one. 
it will be as exciting as the responses you are about to send us can make it. 
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Appendix II.B,2 



SCENARIOS 

ONE: 

Seaman Jones checks into his local CAIFAC (Computer Assisted Instruction Facility) 
for a lesson on the Mark IV, Mod 2, Radar Repeater. Jones is a striker for ET3 (i.e. he is 
trying to make the rating of Electronics Technician Third Class) and he is preparing for the 
ET3 Rating Exam. 

After showing his pass to the guard to ensure that he has the required security 
clearaQce, Jones goes to the nearest available terminal and logs into the ET3 course. He 
selects lesson 21 from the menu. The introduction to the lesson informs him that he will 
need a mockup of the radar repeater for this lesson, Jones goes to the instructional 
equipment area, finds th^ mockup on a rolling cart, and takes it back to the terminal. 
Jones then proceeds with the lesson. 



TWO: 

Journalist Second Class (J02) Bill Brown is studying for the advanced rating exam, 
the JOl/JOC exam. He has logged onto the CAI system aboard the USS CONSTELLATION 
{CVA-64), an aircraft carrier currently on station in the Western Pacific. 

J02 Brown is currently reviewing newspaper editing procedures and has been 
presented with a diagnostic test on photo layout. This test presents him with a story title 
and a number of photographs. Brown's job is to indicate the way he would crop (cut and 
trim) the photos, the captions or cutlines he would write for each photo, and how he would 
arrange the photos and copy blocks (captions or cutlines) on a standard sized newsoaptsr 
page. 

Brown will be using both the light-pen to indicate cropping and layout and the 
keyboard for entering caption text. This lesson is the sixteenth in the preparatory course 
for the JOl/JOC Exam and Brown has also completed the J03/J02 course (35 lessons) as 
well as a special short course on photojournalism (10 lessons). 



THREE: 

The Education Services Officer at Clark AFB, Philippines, is reviewing the training 
records of the men at his command. From this review he must determine what 
aclvancement examinations to requisition from Training Command Headquarters. ThiS 
review is relatively simple, since most of the courses have been CAI anH records have 
been automatically maintained. The ESO can obtain a printout of these records in a 
variety of formats by using an author level program which he learned in a CAI course. 
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After reviewing the records, the ESO can also have notices for all the men 
autofiiatically prepared informing them thaF their rating exam has been ordered. In any 
case the CAI system will automatically record a notice for each student informing him that 
his records have been reviewed, which records were reviewed, when, and by whom. 



FOUR: 

Lieutenant Williams is updating a CAI course on Japan for personnel who are being 
transferred to Japan. In this course the student is introduced to the history, culture, and 
customs of the Japanese. The student is also given information specifically relevant to the 
conduct of American armed forces personnel in Japan. This information ranges from 
status of forces agreement information to sensitive topics such as nuclear power and 
weapons. 

Several thousand personnel ranging from recruits just out of basic training to field 
grade officers (up to Colonel) take thi/course or particular lessons from it each year and, 
it is availabia at all commands and installations throughout the world. Some of the 
information changes frequently and the course has to be updated semi-annually. One© 
Lieutenant Williams finishes' updating the course materials in the courseware at the 
origination point, all future students receive the updated version of the course. 



FIVE; 

Airman Farrel has just reported, along with the other new men in his squadron, to 
the Base Firefighting School. After checking into the school, the men are sent to the CAI 
Termmal Room. A sergeant there gives the men a half-hour lecture with platform 
demonstration on the use of the CAI terminal. The men then go to individual carrells 
where they begin the first of three half-hour lessons on firefighting. They will receive 
lessorTlwo day after tomorrow and lesson three two days after that. Few if any of the 
mAn hav/^ ever seen a CAI terminal, so the sergeant and several of his assistants wander 
room providing individual assistance. 



men have 
about the 



/ 



SIX: 



Technical Sergeant Maxwell Denver had just begun the third phase of his training on 
the ARC 23 Mod 6 Mark XV SSB FSK Teletype. He had been working on the ARC 23 
frequency standard and is now about to start the trouble shooting procedures for the 
emitter follower in the first stage of the demodulator. 

He begins by taking the pretest module, and being assigned a PC board mockup 
which he plugs into his test stand. (Two modules earlier he learned the pecul 'ities of 
the extenders and their test points, and he is using the appropriate extender now.) 
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Denver will be told as he probes the circuit under test the appropriateness of the 
pWnts he selects for viewing on his simulated oscilloscope. If he becomes lost in the hunt 
for the malfunction, he will be prompted and pe^rhaps receive a short review of the basic 
principles he is using in solving the problem. 



SEVEN: 

Staff Sergeant James Kildaire is taking a special CAI course on emergency 
diagnostics as part of his paramedical training. He is proceeding through lesson 14 of the 
30 lesson course. In this la.son he is given a list of symptoms for a patient in a combat 
situation which is also described. Sergeant Kildaire may ask for additional information 
about the patient which he would normally be able to obtain under the given conditions. 
Sergeant Kildaire must then enter his diagnosis of the patient's condition using standard 
medical terminology and suggest emergency treatment procedures. 

In previous lessons he has then received advice from a medical officer with whom he 
was in radio contact. In this lesson, however, he is on his own. The CAI program accepts 
the Sergeant's treatment procedures and then informs the Sergeant of the effects on the 
patient. The Sergeant then recommends further treatment or calls in a medevac team* 
This dialogue is continued until the patient is evacuated or expires. 



EIGHT: 

Staff Sergeant Friendly is preparing a lesson on personnel record keeping in regard 
fo PERSCOM FORM 362-A (REV 1/9/75). The individual serviceman's record of training is 
kept on this form which includes entries pertaining to military training courses, USAFI 
courses, and courses completed in civilian schools as part of the serviceman's career 
enhancement program. 

Sergeant Friendly wants to use Computer Assisted Instruction so that he can 
present a variety of examples that illustrate the basic categories of entries to be made on 
this form. In CAI he can occasionally check to see if the student hdS grasped the basic 
category. If the student has, the CAI can advance him to another category; if not,the 
student can be given remediation until he understands the category. 

At the same time. Sergeant Friendly wants to give the student practice in actually 
making the entries on the form. The serge^^nt cannot decide how best to do this. 



NINE; 

LCDR Moore, the navigator aboard the Polaris submarine USS GEORGE WASHINGTON, 
has just been relieved as Officer of the Deck on the second dog watch. After stopping by 
the galley for a fresh cup of coffee, LCDR Moore goes to the Communications Room. A 
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special stand-alone CAI terminal js kept in the Comm spaces for top secret worK, since the 
Comm spaces are among the most higly secured areas of the submarine. The terminal is 
one of the new stand-alone types developed for use aboard Polans submarines which 
must maintain tight two-way communication silence during patrols. 

LCDR Moore checks out a CAI pack from the duty Classified Materials Control Officer 
and takes the pack to the terminal. Moore plugs the pack in and keys in his personal 
student combination on the terminal to log in and activate the courseware. 

This is the twelfth in a series of thirty lessons on combat command. In this lesson 
LCDR Moore has command of the submarine during a simulated hunter-killer operation. He 
will encounter an enemy submarine of the same type and engage the enemy in combat. In 
his first such simulated engagement, LCDR Moore lost his submarine in the* first five 
minutes of battle. But he has improved through the tutoring of the CAI system and got a 
rating of 750 out of 1000 on his last engagement. 
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Appendix ILC.l 
SPECIAL LETTER TO MILITARY PARTICIPANTS 

[Name of participant] 
[Address of participant] 

Dear [Title and last name]: 



It has been brought to our attention by the University's Office of Contracts and 
Grants that we are restricted from reimbursing government personnel for services out of 
funds from our government contract. Consequently, we will not be able to send you 
tweniy-^ive dollars for responding to the second round of the CAI/CMI study. If you have 
already responded to the second round, thank you. In the event you have not, we hope 
that you will respond promptly in any case. 



Sincerely, 



/s/Monty C. Stanford 

for/ Bill Mann, ISI 

Rick Carlson, ASC 
Tom Martin, ASC 
Monty Stanford, ASC 
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Appendix III.A 
ROUND ONE QUESTIONNAIRE 



You, the experts, have general attitudes that will be contributing to your specific 
responses throughout the study. We need to find out what those attitudes are so we can 
find a common ground if one exists, or at least can better understand your responses. 
The following trade-off questions attempt to tap those attitudes. Place an V next to the 
alternative that best characterizes your opinion and check marks next to the other 
alternatives you would be willing to accept for purposes of consensus. If there are other 
trade-off questions we should have asked, feel free to suggest them. Write us 
explanations if you think it might help. Remember that the context is military CAI/CWI. 
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U The user interface should be 

a) easy to use (even if this means limiting system 

capability) 

b) powerful (even if this makes it hard to use) 

2. The terminal should be 

a) usable for a wide range of tasks (text-editing, 

programming, etc.) 

b) intended specifically for CAI/CMI 

3. The system should 

a) adjust to the user (even if this is expensive 

computerwise) 

b) have the user adjust to it (even if this is expensive 

humanwise) 

4. The user interface should be primarily 

a) built into the hardware 

b) located in the software 

5. The system should be 

a) innovative (even if it sometimes doesn't work) 

b) reliable (even at the expense of discouraging 

innovation) 

6. The terminal/computer resource(s) should be 
•) . stand-alone 

b) time-shared 
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7. The design philosophy should be oriented toward 

a) maximizing benefits (even where costs are high) 

b) minimizing costs (even where benefits are slight) 
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The next fifteen questions deal with ranges of users, tasks, and learning 
environments. Once again place an V nexi to the alternative that best characterizes 
your opinion and a check mark next to alternatives you will accept for purpose s of 
consensus. We have asked you to try to explain what you think each question is getting 
at and how you would like s.ubsequent rounds to probe in greater depth. Feel free to 
write on the backs of pages or to enclose additional pages. 

8. The terminal should be locatable 

a) . in the user's preferred environment 
(even if help is not roadily available) 

b) only where help is readily available 

(even if this is not the user's preferred 
environment) 

9. Military personnel vary greatly in their intellectual 

abilities. Five to ten years from now (1980-1985), the greatest 
payoff will come from user interfaces intended for 

a) the total range of user intellectual competencies 

b) the more intelligent users 

C) users of average intelligence 

10. Military personnel vary greatly in their motivation to leaM. 
Five to ten years from now (1980-1985), the greatest payoff 
will come from user interfaces intended for 

a) the total range of user motivations 

b) the more motivated users 
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c) users of average motivation 

11. User interfaces are needed for many aspects of CAI/CMI activity. 
Five to ten years from now (1980-1985), the greatest payoff 
will come from user interfaces intended for 



a) the total range of system activities 

b) primarily courseware development 

c) primarily research into CAI/CMI usage 

d) primarily student learning 



12. Computerized systems vary greatly in the portion of the 
teaching load that they carry. Five to ten years from now 
(1980-1985), the greatest payoff will come from user interfaces 
intended for 



a) the total range of CAI/CMI 

b) primarily CAI (Where they carry most of the load) 

c) primarily CMI (Whore they carry only part of 

the load) 



13. Military training courses vary greatly in the time it takes for 
students to complete them. Five to ten years from now (1980- 
1985), the greatest payoff will come from user interfaces 
intended for 

a) the total range of course durations 

b) those requiring a week or more 

c) those requiring less than a week 
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14. Military training tasks y/ary\reti\\y in the degree to which 
they have security clearance retirements. Five to ten 
years from now (1980-1985), the Weatest payoff will come 
from user interfaces intended for \ 

a) the total range of security clearances 

b) where the material is unclassified 

c) where the material is classified below secret 

15. Military training tasks vary greatly in the need for spontaneous, 
free-form student responses. Five to ten years from now 
(1980-1985), the greatest payoff will come from user 
interfaces intended for 

a) the total range of user responses 

b) those where spontaneous, free-form responses 

are important 

c) those where spontaneous, free-form 

responses are not important 

16. Military training tasks vary greatly in the need for 
non-canned, student-tailored courseware. Five to ten years 
from now (1980-1985), the greatest payoff will come from 
user interfaces intended for 

a) the total range of student-tailored courseware 

b) those where non-canned, student-tailored course- 
ware is important 
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c) those where non-canned, student-tailored course- 

ware is not important 

17. Military training tasks vary greatly in the need for realistic 
visuals (color, three-dimensions, shading, movement). Five 
to ten years from now (1980-1985), the greatest payoff will 
come from user interfaces intended for 

a) the total range of visuals 

b) where realistic visuals are important 

c) where realistic visuals are not important 

18. Military training environments vary greatly in the access- 
ability of an instructor. Five to ten years from now 
(1980-1985), the greatest payoff will come from user 
interfaces intended for 

a) the total range of instructor accessibility 

b) where instructors are not readily available 

c) where instructors are readily available 

19. Military training environments vary greatly in the access- 
ability of a power supply. Five to ten yers from now 
(1980-1985), the greatest payoff will come from user 
interfaces intended for 

a) the total Vange of power accessabihty 

b) where the power supply must be contained in the 

terminal 
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c) where the power supply is available outside the 

terminal 

20. Military training environments vary greatly in the access- 
ability of communication bandwidth (via lines or 
broadcast frequencies). Five to ten years from now 
{1980-1985), the greatest payoff will come from user 
Interfaces intended for 



a) the total range of bandwidth accessability 

b) where no outside communication is possible 

c) where narrow-band (voice-grade telephone) 
communication is possible 

d) where broad-band (cable-television) 

communication is possible 



21. Military training environments vary greatly in the need 
for lightweight equipment. Five to ten years from now 
(1980-1985), the greatest payoff will come from user 
interfaces intended for 



a) the total range of terminal weights 

b) where the terminal weighs less than thirty 

pounds 

c) where the terminal weighs thirty pounds 



or more 

22. Military training environments vary greatly in the duration 
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of an average learning session. Five to ten years from 
now (1980-1985), the greatest payoff will come fr^m 
user interfaces intended for 

a) the total range of learning session durations 

b) where the learning session lasts less than an hour 

c) where the learning session lasts an hour 

or more 

23. Military training environments vary greatly in the grouping of 
students. Five to ten years from now (1980-1985), the 
greatest payoff will come from user interfaces intended for 

a) the total range of student groupings 

b) where students are in close proximity 

^ c) where students are not in close proximity 
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Ap pendix IILB 
SUMMARY OF ROUND ONE 

So far sixteen of the twenty-five questionnaires have been returned Eight of the 
twenty-three questions reflect consensus, i.e., a single alternative receiving a 
preponderance of the x's and checks. These are represented below with the favored 
alternative first, Ihe second-most favored alternative last, and the response tally m the 
middle. A vote of (10+1 vs 0+2) means that the favored alternative received ten x's and 
one check while the second-most favored alternative received two checks. The eight 
points of consensus are: 

la; easy to use (10+1 vs 1 + 1) powerful 

3a; adjust to user (11 +2 vs 1+0) user adjust 

5b; reliable (9+4 vs 2+0) innovative 

7a; maximrze benefits (8+1 vs 3+1) minimize costs 

8a; in user's preferred environment (11 + 1 vs 3+1) 

where help is available 
lid; primarily student learning (12+0 vs 3+4) 

range of research, coursewriting, and learning 
13a; range of course durations (10+0 vs 3+0) 

duration less than a week 
18b; where instructors are not readily available 

(10+1 vs 3+7) range of instructor availability 
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For eight of the questions, opinions were split between a choice of the total range 
of something and a single one of the alternatives. Theses are represented below with the 
range first, the single alternative last, and the response tally in the middle. 
9; user intellectual competency range (11+2 vs 4+4) 

average intelligence 
10; user motivation range (7+2 vs 4+5) average 

motivation \ 
14; security clearance range (7+2 vs 5+1) unclassified 
15; spontaneous, free-form response range (8+5 vs 6+1) 

is unimportant 
16; non-canned, student-tailored courseware range 

(8+4 vs 8+2) is important 
17; realistic visuals range (7+4 vs 4+3) is important 
19; location of power supply range (5+2 vs 7+0) 

outside termin;>> 
22; length of learning session range (7+2 vs 7+1) less 

than an houi 

For seven of the questions, no pattern of responses emerged. The tallies and 
comments for separate items follow. 
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$ 

General comments and contributor: 

-| continue to be totally mystified by this project. The kind of information that 
seems to interest you is worthless for planning purposes Maybe your second round 
{terminal I/O features) will be more meaningfij!, but i rather expect questions such as 
Vhich is more important, local editing or ^raphics?^ Not that I don't think that reasonable 
questions could be constructed— I just toink you are trying something that is literally 
impossible, in the same sense as violating the second law of thermodynamics.** (Sherwood) 

Td prefer that we not use the word terminal, particularly as it could mean a 
stand-alone system— why not display?** (Berk) 

'•The function of the scenarios is not clear** (Wexler) 

The following entries will deal individually with each first round question and the 
responses: 



****Question One: The user interface should be a) easy to use(even if this 
means limiting system capability)— IQ-x", 1 check — b) powerful (even if this makes 
it hard to use) Tx", 1 check — 

COMMENTS: Neither (a or b)- the interface must be so powerful that it is easy 
to use.(Sherv/ood) "If not easy to use, then at least each activity should be 
self-evident in context. The terminal must not interfer with learning.** 

"These alternatives are not inconf|patible, (Dean) Current software 
developments permit bmhly^pes of user interface to be supported by a single 
system. False dichotomy. (Ford) ^ 

"but more power can be '*uncovered''when needed.** (Zinn) 



****Question two: The terminal should be 

a) usable for a wide range of tasks (text-editing, programming, etc.) S-x", 0 checks— 

b) intended specifically for CAI/CMI. 5'*x**, 2 checks— 

COMMENTS: "The cost benefits of volume production are well known. 
Terminals should not be exclusive military devices.** (Dean) 



♦t**Question three: The system should 

u) adjust to the user(even if this is expensive computer wise) ITx** , 2 checks — b) 
have the user adjust to it (even if this is expensive humanwise) l**x**, 0 checks— 

COMMENT: All systems are adjusted to users and users must adjust 
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themselves to all systems. The question should be one of determining 
tradeoffs for specific situations. (Ford) 



♦♦♦♦Question four: The user interface should be primarily 

a) built into the hardware 5**x", 1. check — b) located in the software B'^x", 2 
check-— 

COMMENTS: "Both about equally-else an imbalanced design" (Sherwood) 

"Question really is one of engineering design. I suspect that 
technology will force the answer. (Dean) 

**This requires a technical background I don't care where it is located as 
long as it works." (Hall) 

"a-When economics are sure to be realized b-Where flexibility should 
be maintained** (Zinn) 



♦♦♦♦Question five: The system should be 

a) innovative (even if it sometimes doesn't work) 2*V*, 0 checks — b) reliable (even 

at the expense of discouraging innovation) 9"x", 4 checks—- 

COMMENTS: "These and similar questions are meaningless. Any real-life 
implementation must strike a balance, and it is not possible even to give an 
inclination in one direction or another." (Sherwood) 

"Innovative during development phases , but reliable when it is put into 
operations with students." (Hall) 

"Quite important (reliability) in operational setting, usually is first 
question asked by military, justifiable or not.*' (Nawrocki) 

"Both- I don't see these as either /or questions. This is like asking if 
you like apples or oranges." (Bork) 



♦♦♦♦Question six: The terminal/computer resource(s) should be 

a) stand alon6 7"x", 3 checks— b) time-shared 6"x", 3 checks 

COMMENTS: a-Where logistics demand it, b-for maximum cost effectiveness 
where possible(Dean) "for CAI/CMI activities-time shared among CAI/CMI 
users, but not with administrative functions." (Hall) 

"former easier to implement given current military structure and 
preferences, though latter may be preferable to reduce inter /intra service 
redundancy." (Nav/rocki) 
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"Most of the time, (stand alone) but with possible access to a 
computer occasionally for computation or large data base." (answered 
reluctantly perhaps, but I am thinking of 10 years from now.) (nawrocki) 

"A blanket recommendation is not possible.** (Ford) 



♦♦♦♦Question seven: The design philosophy should be oriented toward 

* a) maximizing benefits(even where costs are high) 8"x", 1 check— b) minimizing 
costs (even where benefits are slight) 3"x" , 1 check- 
any real-life implementation must strike 

''latter better until benefits can be determined accurately." 

"Some suitable mix." (Bork) 

"The intent will always be to maximize certain benefits and to minimize 
certain costs. Again, requires situation specific tradeoffs. (Ford) 



♦♦♦♦Question eight: The terminal should be locatable 

a) in the user's preferred environment(even if help is not readily available) ITx", 1 
check — b) only where help is readily available (even if this is not the user's 
preferred environment) 3"x", 1 check 

COMMENTS: "except for initial aid, the device and the programs can be made 

to work well with no "help"." (Bork) 



♦♦♦♦Question nme Military personnel vary greatly in their intellectual abilities. Five to ten 
years from now, the greatest payoff will come from user interface intended for: 

a)the total range of user intellectual competencies— H"x", 2 checks-- b)the more 
intelligent users"0"x", 1 check c)users of average intelligence— 4"x", 4 checks 

COMMENTS: "This is basically a question of vocation vs decision skills. Both 
are necessary for maximum benefit, but early emphasis will be on vocation." 
(Nawrocki) "Computer materials can and should be adaptable to a wide 
audience." (Bork) 

"I take payoff to mean a favorable coiv^panson of total 'costs' between 
using skilled human teachersdf available) vs. placing major emphasis on 
CAI/CMI in the following kind of situation; it is necessary to tram a single 
person to a specific level of familiarity or expertise in a subject or on an 
instrument and this is to be done within a prescribed amount of (real) time, 
(not in the flavor of your scenario). The time and proficiency constraints are 
to be jointly satisfied" (Wexler) 
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"most desirable features of terminals seem to serve both b and c." 

(Zinn) 



♦♦♦♦Question ten: Military personnel vary greatly in their motivation to learn. Five to ten 
years from now, the greatest payoff will come from interfaces intended for: 

a)the total range of user motivations-7"x", 2 checks b)the more motivated 
users-Z^x", 0 checks c)users of average motivation— ^''x*', 5 checks 

COMMENTS: ''Instruction s/b written to motivate the user." 
(Weike-Glasser-Miller) 

"Motivation may be an irrelevant question in military environments." 
(Nawrocki) 

The highly motivated individual can learn in a wide variety of 
ways— real learning problems come from those not so motivated." (Bork) 

"The less well motivated." (Anastasio) 



♦♦♦♦Question eleven: User mterfaces are needed for many aspects of CAI/CMI activity. 
...The greatest payoff will come from user interfaces intended for: 

a)the total range of system activities-3"x",4checks b)primarily courseware 
development-0"x",2checks c)primarily research into CAI/CMI usages l"x", 1 check 
d)primarily student learning— 12*x"0checks 

COMMENTS: "tricky question to answer. Interpreted this in terms of ultimate 
goal, though student learning will not be significant withoui others." 
(Nawrocki) 

"small number of terminals of special design(if necessary) could serve b 
and c— (Zinn) 



♦♦♦♦Question twelve: Computerized systems vary greatly in the portion of the teaching 
load that they carry. ...the greatest payoff will come from user interfaces intended for: 

a)the total range of CAI/CMI-8"x",3 checks b)primarfly CAKWhere they carry most of 
the load) 4V, 2 checks c)primarily CMKWhere they carry only part of the load) 
4"x",2 checks 

COMMENTS: "I don't like this, but it seems to be the case. CAI likely to be 
relatively limited to material requiring simulation/gaming, but still hard to say 
with any certainty." (Nawrocki) 

"I regard this as a partially artificial distinction, reflecting our current 
relatively primitive abilities." (Bork) 
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****Que$tion Thirteen: Military training courses vary greatly in the time it takes for 
students to complete them. ...the greatest payoff will come from user interfaces intended 
for: 

a)the total range of course durations lOV, 0 checks b)those requiting a week or 
more-OV, 1 check c)those requiring less than a week— S^x**, 0 checks 

COMMENTS: *'for modules of courses requiring 20 hours or less of study-our 
research indicates that 20 hours is maximum for study length."(Dean) "What is 
magical about a week?"(Bork) 

"user interfaces should respond to rather than determine the aspects I 
have not marked, e.g. if training of short duration is better tailored to 
trainee then terminal should be portable, etc. (no responses given) (Zinn) 



♦♦♦^Question fourteen: Military training tasks vary greatly in the degree to which they 
have security clearance requirements...the greatest payoff will come from user interfaces 
intended for 

a)the total range of security clearances 7V, 2 check b)where the material is 
unclassified S^x", 1 check c)where the material is classified below secret— 0 "x",2 
checks 

COMMENTS: ''b-Classified material would be a management problem." 
(Weike-Glasser-Miller) 

"Mostly because much equipment and info referred to will be classified 
to some extent.''(response a^x") (Nawrocki) 

"Why is it an issue? Is the worry that that the system might not be 
secure?"(Bork) 

"If classified material should be handled in automated system, then 
terminal can designed accordingly to assure securit/(no response indicated) 
(Zinn) 



♦♦♦♦Question fifteen: Military training tasks vary greatly in the need for spontaneous 
free-form student responses, ,Jhe greatest payoff will come from user interfaces 
intended for: a)the total range of user responses-8"x" 5 checks b)those where 
spontaneous, free-form responses are important-2"x", 2 checks c)those where 
spontaneous, free-form responses are not important— 5"x", 1 check 

COMMENTS: "^Education should adapt to student, not vice versa." (Bork) 

"Author of training material should not be constrained, but only advised 
of costs or other considerations which might favor less spontaneous 
response." (Zinn) 
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****Qu8stion sixteen: Military training tasks vary greatly in the need for non-canned , 
student tailored courseware. ...the greatest payoff will come from user interfaces 
intended for: 

a)the total range of student tailored courseware--8 "x",4 checks b)those where 
non-canned, student tailored courseware is important-8V,2 check c)those where 
non-canned, student-tailored courseware is not important-0"x"Ochecks 

COMMENTS: Tm not certain I understand this, any good student-computer 
dialog should provide different experiences for different students." (Bork) 

-Clarify *non-canned* or replace."(Zinn) 



****Question seventeen: Military training tasks vary greatly in the need for realistic 
visuals{color, three-dimensions, shading, movement). ...the greatest payoff will come from 
user interfaces intended for 

a)the total range of visuals-J-x", 4 checks b)Where realistic visuals are important 4 
"x^ Schecks c)where realistic visuals are not important 2 "x", Ochecks 

COMMENTS: "Who know where they are important yet my personal bias is 
that sophisticated interactive graphics directly on a terminal are a luxury with 
low general utility. Peripheral visuals are cheaper generally as effective." 
(Nawrocki) 

"but perhaps experimental work here will show that not all of these are 
equally valuable." (Bork) 

"sometimes line drawings(and animated drawings are more important 
than photos." (Zinn) 



»***Question eighteen: Military training environments vary greatly in the accessability of 
an instructor. ...the greatest payoff will come from user interfaces intended for: 

a)the total range of instructor accessibility 3"x", 7 checks b)where instructors are 
not readily available— 10"x", 1 check c)where instructors are readily available 3"x" 
0 checks 

COMMENTS: "note— Previous answer on this subject was based on the 
knowledge the present CAI systems require some mgt. Ideally future CAI 
systems will require far less management." (V^ike-Glasser-Miller) 

"Opinion? probably use of remote terminals will come into its own." 
(Nawrocki) 

"c— computers may play lesser role."(Zinn) 
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♦^♦♦Question nineteen; Military training environments vary greatly in the accessability of a 
pow©r supply, ...the greatest payoff will come from user interfaces intended for: 

a)the total range of power accessability jrx", 2 check b)where the power supply 
must be contained in the terminal— ^x^ 1 check c)where the power supply is 
available outside the terminal-y^x", 0 checks 

COMMENTS: "doubt this will be feasible in time period required-if so very high 
utility*'(response c«**x*' b«check)--more than likely will emphasize Option c 
with gradual use of option b.(as supplement not alternative)." (Nawrocki) 

"in a true ^cale power will still make considerable demands." (Bork) 

"training decision(no response indicated) (Zinn) 



^♦♦Question twenty: Military training environments vary greatly m the accessability of 
communication bandwidths (via lines or broadcast frequencies). ...the greatest payoff will 
com© from user interfaces intended for: 

a)the total range of bandwidth accessability 6"x", 3 checks b)where no outiide 
communication is possible Tx", 2 checks c)where narrow-band(voice-grade 
telephone) communication is possible 2"x",4 checks d)where broad-band 
(cable-television) communication is possible"3"x",3 checks 

COMMENTS; "this problem does not arise with stand alone systems, one of the 
reasons that such systems will be important in the future." (Bork) 

"c-d "if economical, then interaction with instructors via cable may be 
preferable to computers." (Zinn) 



♦♦♦♦Question twenty-one: Military training environments vary greatly in the need for 
lightweight equipment. ...the greatest payoff will come from user mterfaces intended for: 
a)the total range of terminal weights 6"x", 2 chscks b)where the terminal weights 
less than than thirty pounds— 5"x", 4 checks c)where the terminal weighs thirty 
pounds or more— rx", 2 checks 

COMMENTS: "hard to imagine a terminal of any capacity weighing more than 
30 lbs, ten years from now." (Dean) 

"obviously a technological question. Weight important, but terminal 
effectiveness more important." (Nawrocki) 

"not the most critical issue." (Bork) 

•Portability is not yet an issue" (Anastasio) 
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♦♦♦♦Question twenty-two: Military training environments vary greatly in the du'-ation of an 
average learning session. ...the greatest payoff will come from user interfaces intended 
for: 

a)the total range of learning session duration TyC^ 2 checks b)where the learning 
session lasts less than an hour-y^'x*', 1 checks c)where the learning session lasts an 
hour or more-l^x^O checks 

COMMENTS: ''much more than an hour will bore or at least tire the student." 

(Weike-Glasser-Miller) 

"1-2 hours max." (Dean) ""clearly varies with content of course, need, 
etc. Not a particularly useful question.*'(Nawrocki) 

"HTraining decision " (Zinn) 



♦♦♦♦Question twenty-three; Military training environments vary greatly in the grouping of 
students, -the greatest payoff will come from user interfaces intended for: 

a)thB total range of student groupings 6"x-, 1 check b)where students are in close . 

proximity 4-x^ 3 checks c)where students are not in close proximity — 3*'x-, 2 

checks 

COMMENTS: "students should interact with terminals alone! should discuss 
course material with other students offlineHDean) 

"strictly opinion, current thinking and planning tends to emphasize 
option b/(Nawrocki) 

-small groups of 2-3 offer great advantages advantages for many types 
of work-student learn much from interacting with each other.^CBork) 
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Appendix HLC 
ROUND TWO QUESTIONNAIRE 

Hi, enclosed are the results of round one of the 1980- 1985 military CAI/CMI study 
and the questions for the second round. The response rate was over sixty percent (16 
out of 25) and it took a long time for responses to trickle in. This time we plan to reward 
you (25 dollars) for responding promptly. (If you have not yet returned round one, you 
may want to do so now.) 

Last time we told you that round two would get down to features. Instead we 
decided to use this round as a bridge between the first round and the features round. 
The questions fall into three groups— Whether or not an investment in terminal 
development is likely to lead to the greatest payoff, preparations for round three feature 
identification questions and further probing of round one issues. We will get results to 
you as soon as responses have been collected and analyzed, and we strongly encourage 
yau to be verbose in your responses. Remember that this is more a discussion to help 
you influence us than a reliable sampling of some homogeneous population. 



1. Do you think that an investment in the development of new terminals for 
CAI/CMI will have a significant payoff? What are some of the reasons underlying your 
opinion? 
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2. Do you think that five to ten years from now the mainstream terminal vendors 
will be responsive to the needs of the military CAI/CMI users (assuming no concerted push 
by the military)? What termmal-related needs are likely to go unmet without a concerted 
push? What features do you think will be available by then? 



3. It can be argued that factors unrelated to terminals are responsible for holding 
back the advancement of CAI/CMI. In what areas (other than terminal development) do 
you think that an R&D investment would more significantly advance the 1980-1985 state 
of the art in CAI/CMI? Why? 



It Is likely that the design specifications for CAl/CMl terminals resulting from this 
project will fall Into a hierarchy. At the top level are components that can be 
incorporated at the time of procurement. At the bottom level are components that can be 
incorporated (plugged-in) by the end Ui>er. Components can be input devices, displays, 
storage media, or processors. In the next round we intend to find out your priorities 
regarding which components should be at which levels. Right now we need to find out a 
little about the top through bottom levels. 
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4. What components do you feel could be standard across all terminals? How 
strongly do you feol about each? 



5. What components do you feel should be optional in which situations so that they 
can be added on by the end user. How strongly do you feel about each? 



6. In anticipation of the next round, what specific componentc you want us to 
ask about? Is there any other advice you want to give us? 



As indicated in the summary of the first round, eight questions had responses 
balanced between a total range of something and a particular option. Two of the 
questions related to users of average intelligence and average motivation. We would like 
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to ask those questions again, reworded slightly, to be sure we are reading you correctly. 
Elaborate upon your answers if it will help. 



7. While it is true that the total range of user intellectual capabilities must be 
considered when designing CAI/CMI user interfaces, do you agree that priority should be 
given to interfaces intended for users of average or below average intelligence? Why? 



8. While it is true that the total range of user motivational levels must be 
considered when designing CAI/CMI user interfaces, do you agree that priority should be 
given to interfaces intended for users of average or below average motivation? Why? 



9. What bottlenecks make it difficult for users of average or below average 
Intelligence to use CAI/CMI today? 
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10* What bottlenecks make it difficult for users of average or below average 
motivation to use CAI/CMI today? 



IL In operational terms, what does an "easy to use" terminal look like? (e.g., limited 
choice in options, lack of an alphanumeric keyboard, hardware to lock out features that 
aren^t applicable). 



As indicated in the summary of the first round, a number of the questions elicited 
consensus. The user interface should be reliable, be easy to use, adjust to the user, be in 
his preferred environment, and not be depc^den: ^pOn readily available instructors. We 
need to draw out the implications that flow from these priorities and be certain we are 
reading you correctly. The questions will tackle acquisition of user interface skill and 
support backing up use of the interface. Elaborate upon your answers if it will help. 
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12. How much, when, and primarily from whom/should the beginning user learn 




about the interface? (e.g.,live instructors, other users, remote consultants, software, 
written guides) 



13. How much, when and primarily from whom should the experienced user team 
about additional aspects of the interface? 



lA. What sorts of things should the system be able to take care of so that the user 
does not have to learn about them (e.g.,bad telephone connection, spelling errors, error 
recovery...) 



15. When things are not .'jnctionmg properly and the system cannot assist the 
user, who and what are the p-'m^r;^ sources of support that the user can fall bacK on? 
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16- There was consensus that the greatest payoff will come from an emphasis upon 
student learning (rather than the total range of research, course writing, and learning). 
Are we at a stage where enough is known about course writing and learning so that 
reliable and effective courseware can be developed? 



17. During the first round, a lot of questions about this versus that priority were 
asked. A number of you felt this was unfair, unnecessary, or could not be done. In this 
round we have continued extracting your priorities, preferences, and opinions about areas 
where payoffs are most likely. By now you probably have a personal set of the most 
central priorities when thinking about the 1980-85 military CAI/CMI user interface. What 
are they? 
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APPENDIX IJLD 
ROUND TWO SUMMARY 



1. Do you think an investment in the development of new terminals for CAI/CMI will have 
a significant payoff? What are some of the reasons underlying your opinion? 



COMMENT: Most respondents feel that an investment in terminal development will 
have a significant payoff (Bork, Ditzik, Giunti, Hall, Kimberlin, Rockway, Sherwood, 
Zinn), Reasoning varied, but the following comments are generally representative: 



Yes, but not just terminals — also stand-alone systems with intelligence. 
Existing terminals have been designed primarily for the business market, so 
are not ideal for education. (Bork) 



Yes, if done right. First, in order to have a significant payoff the terminal 
must be general enough for university, public school, industrial, and military 
instruction. However, the entire CAI/CMI system must be applied in such a 
way not to discourage or threaten the instruction by its use. That is the 
system must be adjunct to the normal instruction not a mainline instructional 
system. Plus, the terminal itself must be designed from ground up 
incorporating established human factor requirements. (Ditzik) 



Yes, I do think an investment in the development of new terminals for CAI/CMI 
could have significant payoff. The major reason for this assumption is that 
there are many, particularly CAI type, applications for terminals with varying 
degrees of complexity. These could be designed to match the requirements 
of the particular learning tasks involved and the instructional strategies being 
implemented. For example, the current PLATO IV plasma panel is 
over-designed (and too expensive) for many of the applications for which it is 
being used. Much of the material being delivered is in a programmed 
instructional format with a multiple-choice type response. It would be much 
more economical to use a simple responder augmented by off-line adjunct 
materials. I accept the need for a family of terminals with each class being 
designed to meet a particular set of instructional requirements. Just how 
many classes should be considered and what their functional characteristics 
would be is a matter for a more detailed analysis. (Rockway) 
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COMMENT: Some respondents disagreed (Dean, Ford, Merrill, Ward), reasoning as 
follows: 



No, I believe that commercial vendors will develop competitive terminals in 
response to market forces. With possible exception of Plato, which is still 
unavailable, as far as image and audio are concerned no significant terminal 
has been developed except by industry. (Dean) 



I think existing terminals can do far more than they are being required to do. 
In my view courseware design is far more crucial than design of a particular 
delivery system. (Merrill) 



COMMENT: And two respondents were "lukewarm" about terminal development: 



Strikes me that the available terminals (current) are more than sufficient for 
CMI already and that the only additl' ,al technology with potential utility might 
be audio input/output communication. For CAI, major "need" would seem to 
be low cost sophisticated graphics terminal (3-D, color, etc.).. Also 
recognition of voice or written input might be wprthwhile. The preceding, 
plus increased portability would seem to have highest payoff potential based 
on user "complaints" and to a lesser extent, current data on learning 
processes and man-machine communication. (Nawrocki) 



The PLATO group did tinker and did come up with an interesting terminal 
configuration (although they have yet to meet their original cost projections). 
You ask if similar developments are likely to occur in the near future 
(1980-5). In thinking about the phrase "significant payoff" I find my response 
to be a lukewarm "maybe." (Wexler) 



2. Do you think that five to ten years from now the mainstream terminal vendors will be 
responsive to the needs of the military CAl/CMI users (assuming no concerted push by the 
military)? What terminal-related needs are likely to go unmet without a concerted push? 
What features do you think will be available by then? 
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Among the reasons mentioned were: 

COMMENT: Most respondents feel vendors will not be responsive (7 NO, 1 YES, 3 
N0(?), 3 YES(?). Military and civilian CAI/CMI needs are essentially similar (Bork, 
Dilzik, Ford). Neither market is large enough to influence vendors, who respond 
more to business applications of terminals (Bork, Hall, Zinn). 



Equipment manufacturers will not be responsive to the needs of military 
CAI/CMI users unless a concerted push is made. Their approach in the past 
has been to give educators the very best station that the bankers and airlines 
need for their applications and let the educators adjust their instructional 
materials to fit the existing hardware. The military has very specialized 
training requirements which must be met by a specially designed terminal, e.g., 
the display of electronic diagrams, symbols, mathematic equations, special 
symbols for physics, and chemistry. Terminals must be capable of displaying 
any material which can be printed on a page, display photographic materials 
under systems control, and play audio material under systems control. These 
needs are likely to go unmet because banks and airlines don't need them. 
Unless a concerted push is made it is likely that only the display of upper and 
lower case characters will be available with perhaps some very rudimentary 
microfiche retrieval procedures because bankers need that to check current 
balances before accepting personal checks. (Hall) 



COMMENT: Furthermore, military CAI/CMI terminal needs are not well defined or 
expressed (Nawrocki, Ward). However, some respondents feel that a concerted 
push from the military could influence vendors (Bork, Ditzik, Rockway). 



Vendors will be responsive if there is an economic payoff. The military 
should continue to sponsor R and D to lower costs and must make volume 
purchases. Obviously, agreements and standardizations regarding military 
terminals could result in larger volume buys. (Rockway) 



3. It can be argued that factors unrelated to terminals are responsible for holding back 
the advancement of CAI/CMI. In what areas (other than terminal development) do you 
think that an R and D investment would more significantly advance the 1980-1985 state of 
tho art in CAI/CMI? Why? 
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COMMENT: Almost all respondents stressed need for performance demonstration of 
CAI/CMI and need for emphasis on the total learning environment. Many 
respondents noted that work on utilizing potential of CAI/CMI, especially interactive 
programs, and research on learning/instruction techniques are more important than 
R and D of terminals* Typical responses follow: 



This was addressed in the Educom conference of several years ago, and 
discussed in the report of that conference. The single major factor, in my 
opinion, is the scarcity of good highly interactive learning programs, rewritten 
many times on the basis of extensive student use. Partially this is just a lack 
of experience — we have much to learn about how to write very effective 
student — computer dialogue, and this learning must involve not only research 
but considerable experience. Most of the current authors are still in a 
"textbook" mode, not exploiting the full capability of the media. Much 
additional work, too, is needed with authoring systems which ease the task of 
prisparation of materials. (Bork) 



My advice is "save your money" rather than R and D. I would like to see 
production experiments built around state of the art gear conducted. (Dean) 



One of the biggest hurdles in advancing CAI/CMI is providing authoring 
procedures which are simplified yet powerful enough to allow very 
sophisticated instructional strategies to be employed by content specialists 
who are not high-powered programming specialists. A specialized authoring 
facility could be developed to generate input data which could then be 
compiled into the operating language of any operational CAI/CMI system. This 
would enhance the transportability of curriculum from one system to 'another. 
(Hall) 



Courseware design, strategy, content analysis. In my view:Delivery system 
(terminal) modification makes the least^ difference to learning while 
modifications in strategy and content structure can make major differences. 
Far too much attention to the "cosmetics" of instruction. Far too little effort 
on the substance of instruction. (Merrill) ' 



The broader application of CAI/CMI in the immediate future is largely 
dependent on successful (that it cost-effective) demonstrations. The longer 
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term success of the area would appear to depend on improvements in 
instructional technology as well as reductions in the cost of CAI/CMI 
hardware. On the instructional technology side, one of the major 
requirements is the improvement in instructional strategies to capitalize on 
the flexibility of computerized delivery systems. Another is to develop 
authoring aids to reduce the time and cost of course design and instructional 
materials development. However, even if no improvements were made in the 
current state-of-the-art of instructional technology, simply lowering the cost 
of CAI/CMI systems to make them competitive with conventional techniques 
would do as much as anything to expand applications. (Rockway) 



R and D funding might better be directed towards the development of 
"intelligent" teaching systems for both students and teachers. ^ 

From the point of viev/ of a student working in a conventional CAI/CMI 
environment there simply is not the richness or flexibility that arises in a live 
human teaching environment. The type of adaptation exhibited by a teacher 
who notices the pattern of a student's responses and utterances, and then 
proceeds to make appropriate alterations in a curriculum sequence is only 
clumsily replicated. This lack of response options obviously arises from the 
difficulty (impossibility) of foreseeing or pursuing all the threads emanating 
from a standard curriculum strand. Perhaps one way to improve the situation 
IS to provide the course author/teacher with an ''intelligent teacher's aid." 

The situation might be imagined where a teacher preparing a course can turn 
to a nearby aid and remark that "this material involves concepts CI, C2....and 

has features Fl, F2, and is in the following general relationships with what 

has and will be covered Rl, R2,.... The aid might also ask questions when 
something unusual appeared such as: did you really mean to associate those 
concepts? Are not these features incompatible? Is not this combination of 
relationships curious? etc. Then let the teacher leave and let the teaching aid 
assume the responsibility of interacting with a student when difficulties arise 
in accordance with the directives and associations, supplied by the teacher. 
Thus the aid should make strategic (and "intelligent") use of the (possibly 
loose) information provided by the teacher. It may simplify course 
preparation and expand the range of treatable situations. (The above 1 
description is awkward and needs refining but hopefully indicates the trend of 
my thoughts.) (Wexler) 



Some other possibilities, but probably wouldn't do any more than very 
significant developments in terminals, since the interface with the user affects 
all components: Instructional science, particularly to make better use of the 
dynamic nature of the computer-based training system, including attention to 
the development of learning, self-testing and other skills in the learner. 
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Prescriptions for effective training materials, in order that the first drafl can 
come closer to the final product (cost savings, primarily) 

Incorporation of training into performance systems, including use of 
simulations for practice (what degree of abstraction, fidelity, tutoring, otc,?) 
and monitors of performance in actual operating systems. (Zinn) 



4. What components do you feel could be standard across all terminals? How strongi/ do 
you feel about each? 



COMMENT : Numbers of responses follow each feature. 

CRT Display — 2 

Graphic Display 4 

Still Images — 5 

Hardcopy Output — 2 

Interface to Equipment — 2 

Keyboard — 7 

Pointing Input — 5 

Stored Audio Output — 4 



The standard basic terminal should provide the following features: Display of 
any material capable of being presented on a standard textbook page which 
includes use defined graphics and special characters, keyboard input, light pen 
or touch sensitive input, random access photographic image retrieval and 
display, and random access audio retrieval and display. I would not tolerate 
any deviation on having these facilities available at each station. Making them 
a requirement on all stations has two advantages: (1) it encourages authors to 
develop more sophisticated and richer instructional materials because the 
facilities are there. (2) Because the/ are available at each station they are 
less expensive than if they were produced in smaller quantities. (Hall) 



I don't favor plug-in media — things tend to be written primarily for minimal 
system, so most programs would not use such plug-in facilities. Simple 
graphic input should be standard (like that in TEK 4010s) (Bork) 
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5. \yhat components do you feel should be optional in which situations so that they can 
be added on by the end user. How strongly do you feel about each? 



COMMENT: Numbers of responses follow each feature. 



Still Images — 2 
Hardcopy Output — 3 
Stored Audio Output — 2 
Interface to Equipment — 6 
Videotape — 3 
Large Area Screens — 3 
Pointing Input — 5 
Audio Input — 5 
Special Keyboards — 2 
Computing Power — 2 



6, In anticipation of the next round, what specific components do you want us to asK 
about? Is there any other advice you want to give us? 



COMMENT: Most comments were similar to Hairs. 



I would hope that the next round of questions would include each of the 
components that I mentioned as being requirements for each station to find 
out to what extent my opinions are held generally throughout the survey 
group. (Hall) 



Those Tve already mentioned Avoid ^^terminal" see if the word "military 
makes any difference. (Bork) 
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Ask for specific types of terminal usage anticipated, extent of usage and 
groups using terminals asking about terminal components is wrong. You want 
to determine usage requirements for training, counseling, management, etc., 
and then develop a system which will meet those requirements. (Ford) 



Ask about all. Perhaps you should ask for cost estimates in 1980-5 for 
individual components — which might reflect their plausibility along with 
estimates on what will constitute a reasonable terminal cost at that time (In 
1975 dollars?) (Wexler) 



7. While it is true that the total range of user intellectual capabilities must be considered 
when designing CAI/CMI user interfaces, do you agree that priority should be given to 
interfaces intended for users of average or below average intelligence? Why? 



COMMENT: Consensus was achieved (10 Yes, 3 No, 1 Other). Most comments were 
similar to Wexler's: 



Yes, they will probably constitute the largest satibfiable group, I don't expect 
a great increase in the level of sophistication of interactive dialogues and I 
expect brighter students to realize that an alternative information source (e.g., 
a well-written book on the subject) may be a more efficient use of their time 
{although I expect them to be able to tolerate CAI/CMI), Thus the average or 
below average group may accept more readily the quality of instruction they 
receive. (Wexler) 



Great consideration should be given to designing interface devices which are 
adapted to humans rather than forcing humans to adapt to the devices. 
Devices which are designed for individuals with below average intelligence 
can readily be used by individuals of normal intelligence but the reverse is 
not necessarily true. By designing for lower intelligence individuals the 
terminals will be available to a larger audience. This is especially important 
in the military where a broad spectrum of individual differences must be 
accommodated. (Hall) 



Policy Study on CAI Terminals 76 
Appendix III.O - Round Two Summary 



No. Terminal should be flexible enough to be generally useful. The cliche 
(with a little data to support it) is that bright students learn no matter what 
the instructional treatment, but that average and below average students 
need special attention. However, this more a problem of courseware 
development and implementation priorities than of terminal design. (Ford.) 



8. While it is true that the total range of user motivational levels must be considered 
when designing CAI/CMI user interfaces, do you agree that priority should be given to 
interfaces intended for users of average or below average motivation? Why? 



COMMENT: Consensus was not achieved (6 Yes, 4 No, 3 Other). Representative 
responses follow: 



Yes, the high motivation trainees will acquire information and skills by other 
means than through CAI and CMI. That is, computer assistance of this Kind is 
less important for highly motivated trainees. Some skills practice which is 
particularly aided by computing (e.g., highly realistic simulations) may be 
important to all trainees, and particularly the motivated ones who may be 
expected to do especially well on the job. (Zinn) 



Motivation is a bag of worms. Stay out of it in considering design of CAI/CMI 
user interfaces. Nobody knows what will work with a particular student 
under specified conditions at any particular time. (Ward) 



No. See question 7. Motivation is not a simple trait — different students are 
motivated by different treatments. Those students who are motivated by 
CAI/CMI should probably be given priority for using it. (Ford) 
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9. What bottlenecks make it difficult for users of average or below average intelligence 
to us© CAI/CMI today? 



COWfvOT: Reliance on alphanumeric rather than graphics terminals and consequent 
emphasis on reading comprehension (Dean, Giunti, Kimberlin, Sherwood, Nawrocki). 
Poor courseware (Ditzik, Merrill, Sherwood). Poor learning environments (Bork, Hall, 
Rockway, Wexler, Zinn), e.g.. 



The bottlenecks reside in our ignorance about what things are difficult and 
what things are easy for users of average or below average intelligence. 
(Ford) 



Hardware and software unreliability, difficulty of use, and poor human 
engineering. (Rockway) \ 

{ 

\ 

) 

Keyboard arrangements noS^oJai^us to novice user of keyboards. 
Identification of function keys confusing to novice user. Terminals which lack 
effective pointing capability (light pen, cursor, etc.) Output devices (and 
displays) with limited character sets. Inconvenient editing facility for altering 
text before input (as answer or request) Frustration of slow displays and 
limited line length (and number of lines on screen) (Zinn) 



10. What bottlenecks make it difficult for users of average or below average motivation 
to use CAI/CMI today? 



COMMENT: Most respondents referred to response for question 9, or made similar 
comments. However, the concept of rewards for CAl/CMI experiences surfaced 
here: 



Bottlenecks for average and below average motivated students are: A. Their 
lack of experience and training in a self-paced environment where the burden 
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is placed on them to teach themselves rather than to rely on a teacher to toll 
them what they should know. B. Their inability to relate current success to 
future tasks. C. The aloneness a student can feel in a self-paced class 
situation. D. The demotivating environment a military student might find 
outside the classroom. E. The possibility of an unwelcome assignment after 
the course is completed. (Kimberlin) 



11. In operational terms, what does an "easy to use" terminal look like? (e.g., limited 
choice in options, lack of an alphanumeric keyboard, hardware to lock out features that 
aren^t applicable). 



No long arrays of mysterious buttons with Cryptic labels. No visible controls 
which the user should not use. Off-on switch clearly visible. (Bork) 



Not sure that this is a termina' question except that clutter should be avoided 
and operational features shojid be obvious to the user — i.e., see a good 
office copier — it is obvious how to use it. (Dean) 



12. How much, when, and primarily from whom should the beginning user learn about the 
interface? (e.g., live instructors, other users, remote consultants, software, written guides.) 



COMMENT: Primarily on-line, from the system itself (Ford, Hall, Kimberlin, Merrill, 
Nawrocki). 



Best would be right from system itself (self-instruction) via terminal. All 
other options O.K., but only for special problems.(Nawrocki) 



81 



ERIC 



Policy Study on CAI Terminals 79 
Appendix III.D - Round Two Summary 



COMMENT: Primarily instructors^ more experienced students, or off-line media (Bork, 
Giunti, Ward, Zinn) 



I don't favor widespread use of instructors for this particular task. (Bork) 



At the initial exposure, live instructors should provide as much instruction on 
interfaces as is required to make each student comfortable. (Giunti) 



Live instructors or other users, primarily because this is the method they are 
familiar with. (Ward) 



COMMENT: Both on-line and off-line instruction (Dean, Ditzik, Rockway, Sherwood, 
Wexler). I 



Absolute basics should be presented by a human, and additional instruction 
should be given by the device. (Sherwood) 



Printed guides plus CAI at the terminal should satisfy most learning 
requirements. A human proctor or instructor should be available for 
consultation, either in person or via communication link. (Rockway) 



13. How much, when and primarily from whom should the experienced user learn about 
additional aspects of the interface? 



COMMENT: Most respondents indicated on-line helps, consultants, and/or off-line 
documentation. 
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From on-!i c -ad off-line documentation, and by on-line and off-line 
communication with experienced users. (Sherwood) 



Manuals, perhaps video or slide-tape presentation, primarily. Often help (or a 
suggestion) may come from another user. For complex tasks, human aid 
(experienced colleague or a live instructor) is important. (Zinn) 



14. What sorts of things should the system be able to take care of so that the user does 
not have to learn ^bout them (e.g., bad telephone connection, spelling errors, error 
recovery.) ^ 



COMMENT: Respondents seemed to take the word "system" m this question to heart. 
For example, 



The system should include all the resources (even human) to take care of the 
problems identified, (Rockway) 



COMMENT: While most respondents seemed m basic agreement with Dr. Rockway, 
Dr. Wexler added some details: 



The system should handle hardware errors and should indicate (e.g., by 
turning on a red4ight) that it is trying to do something. The light is turned 
off/to green when the difficulty is resolved. A prolonged red might lead a 
user to try the CONNECT - SIGN-ON - RESTART sequence. Softer errors 
remain in the user's bahwick (e.g., spelling, referencing an unknown file, etc), 
although the system should try to make plausible guesses about the user's 
intent and inditate its hypothesis prior to carrying out the action. (Wexler) 
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15. When things are not functioning properly and the system cannot assist the user, who 
and what are the primary sources of support that the user can fall back on? 



COMMENT: Most respondents said support staff, as in the following comments: 



A fall back program should consist of a prepared package of study guides and 
references that relate to each lesson or portion of the lesson that is on the 
computer. This may be considered redundant, but the same package along 
with the off-line material will be a self-paced course program that may serve 
after a student has left the school, or serve in those areas where a terminal 
is not available. The Vho** may be several different people, depending upon 
the situation. In a unit, it may be a supervisor, peer, or in the worse case, 
the man may have to ''dig'* himself out. In a formal training environment, it 
will be the class instructor. (Kimberlin) 



The system should have a manual backup in the form of instructor support 
and some conventional media to handle automated system failures. i(Rockway) 



Whan the terminal environment becomes a malfunctioning environment (*7our 
terminal is working, why isn't mine?**) the normal sources of support would be: 

1. A telephone call to the center (unanswered?) 

2. Nearby users (unavailable?) 

3. A written terminal guide. (Wexler) 



This seems a training decision, not a hardware design consideration. But 

When the system fails it should recover as much as possible automatically, 
perhaps prompting the student for information needed to restart. When the 
system canM handle this but is still live, it should offer the trainee information 
which may be helpful for him to initiate recovery. When all else fails, a 

written guide of "What to do if ** should be handy to the terminal; in many 

cases it can get a user going again with minimum delay and embarrassment. 
Then human aid should be at hand when all else fails! (Zinn) 
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16. There was consensus that the greatest payoff will come from an emphasis upon 
student learning (Rather that the total range of research, course writing, and learning). 
Are we at a stage where enough is known about course writing and learning so that 
raliable and effective courseware can be developed? 



COMMENT: No consensus was reached (8 Yes, 6 No). Representative responses 
follow: 

No, much more study must be done in the areas of what is good material for 
CAI lesson, what are best strategies, what is best for different types of 
students, what authoring techniques in teams are most efficient and 
productive. This area has no firm data - everyone seems to do their own pet 
thing and ignore all oiher*s efforts, (Giunti) 



There is still a great deal that we need to know about course writing and 
learning so that reliable and effective course work can be developed. 
Perhaps one strategy of accomplishing this is to develop course material and 
then examine it carefully to see what features of it have contributed to the 
learning. Repertories of alternative procedures and techniques need to be 
developed and examined in view of learning difficulties encountered by 
students so that course material can be improved (Hall) 



If the other consultants think we are they are very naive as to what is known 
about instruction. I feel we have a tremendous amount left to \earr We 
have just started to learn how to design courses, most are very weak. We 
have many more questions than answers. If someone thinks we know it all, I 
invite them to write for a list of questions I'd like answered. (Merrill) 



Ha! Depends who you talK to! If we knew whal was "effective" and for whom 
and under what conditions, could probably develop a major in "Educational 
Engmeermg"! Despite all the handbooks and texts, courseware development 
remains heavily intuitive. Dialogue systems where student selects 
instructional style and material (Socratic) will impact heavily on solving this 
problem. (Nawrocki) 



I think so, although it is unlikely that good courseware can be proven to be 
good by actual measurement — there are too many variables. (Sherwood) 
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Yes - especially m the sophisticated instructional development models used by 
the military. See Army Regulation 350-100-1 (1968 version) - Systems 
Engineering and Training for e<ample. (Ward) 



No, not in general, although some on-gomg projects may take exception. It is 
very well to specify a set of terminal behaviors in a course and then try to 
carefully structure *he course to achieve them. However I seem to remain 
unimpressed with the quality of objectives or rather the "interpretations" that 
have been made of them. On the other hand, by a suitable redefinition of 
competency level, much existing courseware could certainly be deemed 
"reliable and effective." (Wexler) 



17. During the first round, a lot of questions about this versus that priority were asked. 
A number of you felt this was unfair, unnecessary, or could not be done. In this round we 
have continued extracting your priorities, preferences, and opinions about areas where 
payoffs are most likely. By now you probably have a personal set of the most central 
priorities when thinking about the 1980-85 military CAI/CMI user interface. What are 
they? 



COMMENT: Although most responses to this question repeated responses to 
previous questions, Giunti and Klmberlin emphasized transportability as a central 
priority: 



I would also like to see a central pomt for preparation of common lesson 
material so that all Army CAI/CMI schools would teach or use identical 
programs ftr instruction. This would of course require a certain amount of 
commonality or transportability between systems. (Giunti) 



Central priorities revolve around transportability of lesson material. I don'^t 
believe that CAI/CMI will really be able to grow into an accepted media unless 
we are able to reduce the cost. One way of course is to spread the cost 
over many students and this means distribution. Therefore, such Wems as 
language and terminal standardization will provide an early step down the 
road to transportability. (Kimberlin) 
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Appendix III.E 
ROUND THREE QUESTIONNAIRE 

Your 

Name_ ^ = 

Date - 

Please return to: 
T. Martin or M. Stanford 
Annenberg School of Communications 
University of Southern California 
Los Angeles, Ca. 90007 

GENERAL QUESTIONS 



A number of you remarked m your round two responses that you felt 
terminal-related problems were not the area where a concerted push by the 
military would result m the greatest payoff. 



1. Considering the 1980-85 timeframe, how would you rank the following 
investment strategies in lerms of their potential (1 « greatest potential, 4 - 
least potential) for advancing CAI/CMI state of the art^ 

invest in: 

mnovattve CAf^CMI terminal technology 

innovative CAI/CMI pedagogical software 

technology (see the pedagogical software 

needs table below) 

innovative CAI/CMI coursewntmg 

large scale use of existing hardware, 

software, and courseware 
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A number of you indicated m your round two responses that you felt more could be 
gained by not distinguishing between civilian and military CAI/CM! terminal needs. Let s 
see if we can get consensus one way or another. 

2. Do you think that it is in the best interests of advancing the CAI/CMi terminal 
state-of-the-art to combine civilian and military needs as opposed to focusing on just 
military or just civilian needs. 



COMBINE KEEP DISTINCT OTHER 



3. In what respects do you feel that 1980-85 military CAI/CMi terminals should differ 
from civilian ones? (You might want to refer to the table of terminal-related functional 
needs below.) 
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TERMINAL-RELATED QUESTIONS 



4. In the responses to the second round, many features were mentioned as relevant to 
CAI/CML While \\ is natural to talk in terms of features, our terminal experts would much 
rather we pin you down on the functional needs underlying various types of features, 
leaving final feature specifications to them. Consequently, we have developed a list of 
functional needs. They are listed down the left side of the following table. For each 
functional need, we would like three pieces of information from you: 

(1) do you think all CAI/CMI terminals should provide for the need? 



++ definitely yes 
would be nice 
0 neutral 

- probably not 

— definitely not 

(put one of these five codes m each row of the ALL TERMINALS column.) 



(2) assuming that not all terminals provide for the need, what special situations 
justify the putting together of special terminals' (put situation-justification 
explanations in relevant rows of the EXCEPTION column.) 



(3) how would you restate the functional need »n terms that get closer to what you 
perceive the real need to be^ (put rewordirgs m each row of the REWORDED NEED 
column ) 



Keep track of additional need categories th^t oc'ur to /ou omce questions 5 and 6 will ask 
for them. 
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TRANSIENT VISUAL OUTPUT 

a. A variety of predefined symbol 
sets (3 or more) can be intermixed 
during display. 

b. Special symbol sets are 
programmable when needed. 

c. Simple straight line figures 
(involving few lines) can be 
generated. 

d. Complex line figures (involving 
many lines) can be generated. 

e. Complex figures (involving 
shading and texture) can be 
generated. 

f. Displays may be generated in 
color. 

g. Displays may be generated that 
contain as many as four thousand 
readable symbols. 

h. Stored visuals (with resolution 
equal to that of a television screen) 
can be displayed. 

i. Stored visuals (with resolution 
high enough for reading a picture of 
a college textbook page) can be 
displayed. 

j. Moving visuals can be displayed at 
flicker-free speeds. 



no 
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HARDCOPY VISUAL OUTPUT 

k. Exact reproductions of screen 
images can be produced. 

I. Alphanumeric text, using a single 
type font, can be printed. 



m. Computer -composed spe^h can 
be generated and transmitted to the 
user. 

n. Pre-recorded audio output (equal 
in quality to AM music) can be 
transmitted to the user. 



o. Signals are available for 
controlling external equipment. 

p. Lights under some keys on the 
keyboard can be turned on or off. 



TRANSIENT AUDIO OUTPUT 




OTHER OUTPUT 
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INPUT 

q. Strings of alphanumeric 
characters can be typed in by the 
user. 

r. Frequently invoked functions can 
be specified unambiguously by 
carrying out a single action. 

s. Locations on the screen can be 
specified by touching or pointing. 

t. Lines can be drawn on a 
two-dimensional surface. 

u. A clearly spoken word chosen 
from 8 limited vocabulary (about 20 
words at any one time) can be 
recognized about 902 of the lime. 

V, Signals that are equivalent to 
terminal-entered input can be 
received from devices plugged mlo 
the terminal. 

OTHER TERMINAL FEATURES 

w. The stand-alone terminal has 
sufficient processing capability to 
manage instruction 707. of the time 

X. Digital and analog memory are 
available within the terminal. 
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5. Now that you have seen the twenty-four functions, what additional needs come to mmd 
for which you strongly feel features should be standard across all terminals'^ 



6. What additional features do /ou feel should be de^^eloped for specia' situations'^ Again 
we would appreciate situation-justification explanations. 
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PEDAGOGICAL SOFTWARE QUESTIONS 

7. In the responses to the second round, many of you mentioned that there was a need 
for a more adaptis^e, intelligent interplay between student and courseware or teacher and 
coursewnter. Since there are so many things you might have had m mind, and since it 
appeared so frequently, we felt it was necessary to probe the area in greater depth. A 
list of software functional needs has been developed similar to the terminal functional 
needs list. This time we are asking for four pieces of information about each need: 

(1) assuming that by 1980-85 it is possible, do you think all CAI/CMI systems should 
provide for the need? 

++ definitely yes 
+ would be nice 
0 neutral 

- probably not 

— definitely not 

(put one of these five codes in each row of the ALL SYSTEMS column.) 

(2) assuming that by 1980-85 i* is possible, but that not all systems provide for the 
need, what special situations justify the putting together of special software? (put 
situation - justification explanations in relevant rows of the EXCEPTION column.) 

(3) how would you restate the functional need in terms that get closer to what you 
perceive the real need {or needs) to be^ (put rewordmgs in each row of the 
REWORDED NEED column.) 

(4) how great a payoff do you anticipate from an investment now m software that 
will respond to the need by 1980-85? 

++ very high payoff anticipated 

+ high payoff 

0 moderate payoff 

- low payoft 

— no payO'f anticipated 
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STUDENT-RELATED SOFTWARE 
The system can synthesize: 

a. Instructional sequences that are 
tailored to the abilities and/or 
weaknesses of individual students. 

b. Problems and examples that 
respond to the interests of particular 
students. 

c. Hints that reduce the difficulty of 
problems. 

d. Summaries of a student's 
progress throughout the course. 

The system can follow and respond 
meaningfully to: 

e. Course-related problem-solving 
strategies ^{even though novel) 
empfoyed by students. 

f. Course-independent (but 
pedagogically relevant) 
problem-solving strategies employed 
by students. 

g. Pauses due to a student's inability 
to decide what to do next. 

h. Course-related quest<oris or 
statements typed in by students 
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i. Course-independent (but 
pedagogically relevant) questions 
typed in by students. 

j. Dialog cues (i.e., topic shifts, 
impatience, bewilderment, wandering 
attention). 

TEACHER-RELATED SOFTWARE 

The system can: 

k. Derive course-related strategies 
from examples provided by the 
instructor. 

I Accumulate course-related 
concepts (with associated vocabulary) 
/Tom examples provided by the 
instructor. 

m. Contrast concept/strategy 
information with information about 
potential students, detect difficulties, 
and advise the instructor of the 
difficulties. 

n. Discover patterns of 
course-related behavior and advise 
the instructor during the course. 
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8. Now that you have seen the fourteen software functions, what additional needs come 
to mind for which you strongly feel filatures should be standard across all systems? 



9, What additional software features do you feel should be developed for special 
situations? Again we would appreciate situation - justification explanations. 
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Ap pendix III.F 
ROUND THREE SUMMARY 



1. Considering the 1980-85 time frame, how would you rank the following investment 
strategies in [em's of their potential (1 « greatest potential, 4 = least potential) for 
advancing CAI/CMI state of the art? 

Invest in: 

111111222xx34 Innovative CAI/CMI pedagogical 

software technology 
1111122223334 Innovative CAI/CMI coursewriting 
1122x33444444 Innovative CAI/CMI terminal technology 
22x3333344444 Large scale use of existing hardware 

software, and courseware 

(x stands for an averaged rating of 2.5) 



2. Do you think that it is in the best interests of advancing the CAI/CMI terminal 
state-of-the-art to combine civilian and military needs as opposed to focusing on just 
military or just civilian needs. 



Combine —10-- Keep Distinct —0— Other —2— No Response — 2— 



<The two •'other'* responses indicated that needs could be combined except that in some 
situations special terminals would be needed. The comments have therefore been 
incorporated into the responses to question three.) 



3- In what respects do you feel that 1980-85 military CAI/CMI terminals should differ 
from civilian ones? 

No differences 4 
No response 3 
Rugged construction to enable use 
in hostile environments 5 
Greater emphasis on nonverbal 

communication (ie. audio, 

video, graphics, i^on-keyboard 

input) 3 
Greater need for stand-alone systems 1 
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Do you think all CAI/CMI terminals should provide for the 
following functional needs? 

.++ definitely yes (shown below as Vr) 
+ would be nice 
B neutral 

probably not 
— definitely not (shown below as «) 

no response (shown below as o) 



[The sections below were reordered in an informal order of 
positiveness of response, for the participants' ease of 
reading, as shown.] 



VnVVo*o*oVVoV>*oVVnV++ q. Strings of alphanumeric characters 

can be typed by the user. 

VrtVVoVVoVVfVVVfVnV++8 r. Frequently invoked functions can be 

specified unambiguously by carrying 
out a single action. 



vnWovvrVnVVrtV>v+++- b. Special symbol sets are 

programmable when needed. 
>vvoVVnVVo*oV+++00« a. A variety of predefined symbol 

sets (3 or more) can be intermixed 

during display. 
Vo*nVVoVVn*oV+++o0« c Simple straight line figures 

(involving few lines) can be generated. 
Vf^vvnVVnVVff +++00- 0. Si gna I s are ava i I ab I e for 

controlling external equipment. 
VnVVo'oVVnV++++00- s. Locations on the screen can be 

specified by touching or pointing. 



>vVoVVoV++++++0— v. Signals that are equivalent to terminal - 

enteredinput can be received from 
devices plugged into the terminal. 
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>WnHn'nV+++++00«« Stored visuals (with resolution equal to 

that of a television scree'n) can be 
di splayed. 

iVVnViWnV++++0~« L Alphanumeric text, using a single type 

font, can be printed. 
>vvnV++++++++00« t. Lines can be drawn on a two-dimensional 

surface. 

>vvnWnV+++++0 — ■ d. Complex line figures (involving many 

lines) can be generated. 
VnWnV++++++00-« i. Stored visuals (with resolution high 

enough for reading a picture of a 

college textbook) can be displayed. 
Vo*oV++++++0000- w. The stand-alone terminal has 

sufficient processing capability 

to manage instruction 70% of the time. 
VoWoV++++++0 — « n. Pre-recorded audio output (equal 

in quality to AM music) can be 

transmitted to the user. 
Vhv+++++++++0-» j. Moving visuals can be displayed 

at flicker-free speeds. 



VnV+++++++000«» f. Displays can be generated in color. 

VV+++++++O000 — m. Computer-composed speech can 

be generated and transmitted 

to the user. 

V5firnV++++000 p. Lights under some keys on the 

keyboard can be turned on or off. 

V!f++++++00000 — X. Digital and analog memory are 

available within the terminal. 

Vn'nV+++++o0 — «■ k. Exact reproductions of screen 

images can be produced. 



>v+-f++00000-««« e.' Complex figures (involving shading and 

texture) can be generated* 

+++++++00 — u. A clearly spoken word chosen from 

a limited vocabulary (about 20 words 
at any one time) can be recognized 
about 90% of the time* 

+++++0000 — g. Displays can be generated that contain 

as many as four thousand readable 
symbols. 

4(2,3) Questions regarding special situations justifying special 
terminals and restated functional needs. 
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Seven of the 14 respondents made no comments. 

Of the remaining seven, two or more people made the same comment 

only three times. 

Whether or not terminals or software systems should provide the 
features is an empirical question based on the feature's 
demonstrated instructional effectiveness (or lack thereof). 

WARD, NAWROCKI 

Primary need is for digital, not analog memory. 

DEAN, DITZIK 

Exact reproductions of screen images can be produced on demand 
at a single station within the terminal area to serve all users 
- but not on every terminal. 

HALL, DITZIK, KIMBERLIN 

5 and 6 — Questions regarding additional needs and features. 

Seven of the 14 respondents made no comments. 

Of the remaining seven, in no case did two or more people suggest 

the same feature. 

7{1). Do you think all CAI/CMI systems should provide for the 
following needs? 



++ definitely yes (shown below as >v) 
+ would be nice 
0 neutral 

probably not 
— definitely not (shown below as «) 

no response (shown below as o) 

^The sections below were reordered in an informal order of 
positivfiness of response, for the participants* ease of reading, as 
shown.] 

>v>vvo*nvvn*nvvnv++oo a. Instructional sequences that are 

tailored to the abilities and/or 
weaknesses of individual students. 

VnVVoVVnV++++++oo b. Problems and examples that respond 
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to the interests of particular 
students. 

Vo*«*oV+++++++++0 c. Hints that reduce the difficulties 

of problems. 

VoWn*hv++++++8» d. Summaries of a student's progress 

throughout the course. 



VhVVhV+++++++o8- 



>VVoV>V+++++++oB- 

WWWW++++++O00- 

VoWoV++++++oB0- 

^o*o*oV+++++oBBB- 



m. Contrast concept/strategy 

information with information about 
potential students, detect 
di fficul ties, and advise the instructor 
of the di f f icul ties. 

n. Discover patterns of course- 
related behavior and advise the 
instructor during the course. 

e. Responsive to course-related 
problem-solving strategies (even 
though novel) employed by students. 

k. Derive course-related strategies from 
examples provided by the instructor. 

h. Respond to course-related questions 
or statements typed in by students. 

I. Accumulate course-related concepts 
(with associated vocabulary) from 
examples provided by the instructor. 



^v^vy«vVf+ooB08B0B g. Respond to pauses due to a student's 

inability to decide what to do next. 
V0V++++++08BBB- f. Respond to course-independent (but 

pedagogical ly relevant) problem-solving 

strategies employed by students. 
>V++++++++o8BBB« j. Respond to dialog cues (eg. topic 

shifts, impatience, bewilderment 

wandering attention). 
^V+++++o0BBB8-» i. Respond to course- independent (but 

pedagogical ly relevant) questions 

typed in by students. 
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7(2 and 3) Questions regarding special systems justifying special 
software and restated functional needs. 

Nine of the 14 made no comments. 

None of the remaining 5 made equivalent comments. 

7(4} Question regarding payoff ratings for software development. 

Ratings in this column were for the most part identical to 7(1) 
ratings and so are not repeated here. 

8 and 9 Questions regarding additional needs and additional software 
features. 

Nine of the 14 made no comments. 

Two of the remaining 5 mentioned that portability across systems 
is also highly desirable. 

HALL, ROCKWAY 
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